Aug 04 17:59:51.956889 ip-10-0-139-201 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 17:59:52.332040 ip-10-0-139-201 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:52.332040 ip-10-0-139-201 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 17:59:52.332040 ip-10-0-139-201 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:52.332040 ip-10-0-139-201 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 17:59:52.332040 ip-10-0-139-201 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:52.333832 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.333716    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 17:59:52.338323 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338308    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:52.338323 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338324    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338329    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338332    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338336    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338339    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338342    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338346    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338349    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338352    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338355    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338358    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338360    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338363    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338366    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338369    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338372    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338374    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338377    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338380    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338383    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:52.338389 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338386    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338389    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338391    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338394    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338397    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338400    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338403    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338406    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338408    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338411    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338413    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338416    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338418    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338423    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338426    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338429    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338432    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338435    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338438    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:52.338860 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338441    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338444    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338446    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338449    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338452    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338455    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338458    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338460    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338463    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338465    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338468    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338470    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338473    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338475    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338478    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338481    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338484    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338486    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338489    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338492    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:52.339308 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338494    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338497    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338500    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338502    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338505    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338509    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338512    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338514    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338517    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338519    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338522    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338526    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338530    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338533    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338536    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338538    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338541    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338543    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338546    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338548    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:52.339801 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338551    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338554    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338556    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338559    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338561    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338564    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338568    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338571    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338573    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338576    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338579    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338581    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338584    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338586    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338589    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338591    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338593    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338596    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.338612    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:52.340282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340011    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340020    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340024    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340028    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340031    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340034    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340036    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340039    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340042    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340044    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340047    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340049    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340052    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340055    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340057    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340060    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340063    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340066    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340069    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340072    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:52.340765 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340075    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340078    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340080    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340083    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340086    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340088    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340091    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340093    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340096    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340098    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340101    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340103    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340106    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340108    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340111    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340113    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340116    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340119    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340122    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340125    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:52.341317 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340127    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340130    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340133    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340135    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340138    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340141    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340143    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340146    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340149    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340152    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340156    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340160    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340163    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340166    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340169    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340172    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340175    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340177    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340180    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:52.341822 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340182    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340185    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340188    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340191    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340195    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340198    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340201    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340204    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340206    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340209    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340211    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340214    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340217    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340221    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340223    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340226    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340228    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340231    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340234    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:52.342280 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340236    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340239    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340242    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340244    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340247    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340249    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340252    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340254    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340257    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340260    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340262    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340265    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340267    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340270    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340273    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340275    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340278    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340280    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340283    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340286    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340289    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:52.342745 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340670    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340675    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340677    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340680    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340683    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340686    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340689    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340692    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340695    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340698    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340700    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340703    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340705    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340708    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340711    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340714    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340717    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340719    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340722    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340725    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:52.343259 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340727    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340730    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340732    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340735    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340738    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340740    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340743    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340746    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340748    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340751    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340754    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340756    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340759    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340761    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340764    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340766    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340769    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340773    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340775    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340778    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:52.343751 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340781    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340787    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340790    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340793    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340795    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340798    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340800    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340803    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340805    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340808    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340811    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340814    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340818    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340821    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340823    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340827    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340829    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340832    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340834    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:52.344247 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340837    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340839    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340842    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340845    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340847    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340850    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340853    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340855    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340858    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340860    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340862    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340865    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340868    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340871    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340875    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340878    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340881    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340883    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340886    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340888    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:52.344717 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340891    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340893    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340896    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340899    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340901    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340903    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340906    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340908    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340911    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340914    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340916    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340921    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340924    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340928    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340931    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340933    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340937    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340941    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340944    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:52.345200 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.340947    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341022    2581 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341033    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341039    2581 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341043    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341049    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341053    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341058    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341062    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341066    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341069    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341072    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341075    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341079    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341082    2581 flags.go:64] FLAG: --cgroup-root=""
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341085    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341088    2581 flags.go:64] FLAG: --client-ca-file=""
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341091    2581 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341094    2581 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341098    2581 flags.go:64] FLAG: --cluster-domain=""
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341101    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341104    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341108    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341111    2581 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 17:59:52.345681 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341114    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341117    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341122    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341125    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341128    2581 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341131    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341134    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341137    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341140    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341144    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341147    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341150    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341153    2581 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341156    2581 flags.go:64] FLAG: --enable-server="true"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341160    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341165    2581 flags.go:64] FLAG: --event-burst="100"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341168    2581 flags.go:64] FLAG: --event-qps="50"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341171    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341175    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341178    2581 flags.go:64] FLAG: --eviction-hard=""
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341181    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341184    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341187    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341190    2581 flags.go:64] FLAG: --eviction-soft=""
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341193    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 17:59:52.346280 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341196    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341199    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341202    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341204    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341207    2581 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341210    2581 flags.go:64] FLAG: --feature-gates=""
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341213    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341216    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341219    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341224    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341227    2581 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341230    2581 flags.go:64] FLAG: --help="false"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341233    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341236    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341238    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341241    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341244    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341248    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341251    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341254    2581 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341256    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341259    2581 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341264    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341267    2581 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 17:59:52.346908 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341270    2581 flags.go:64] FLAG: --kube-reserved=""
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341273    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341276    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341279    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341282    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341285    2581 flags.go:64] FLAG: --lock-file=""
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341288    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341290    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341293    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341298    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341301    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341304    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341307    2581 flags.go:64] FLAG: --logging-format="text"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341309    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341313    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341316    2581 flags.go:64] FLAG: --manifest-url=""
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341319    2581 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341325    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341328    2581 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341332    2581 flags.go:64] FLAG: --max-pods="110"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341335    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341338    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341340    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341343    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341347    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 17:59:52.347490 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341350    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341352    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341359    2581 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341362    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341365    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341369    2581 flags.go:64] FLAG: --pod-cidr=""
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341372    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341378    2581 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341380    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341383    2581 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341386    2581 flags.go:64] FLAG: --port="10250"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341389    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341392    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c02904ec54a31d64"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341395    2581 flags.go:64] FLAG: --qos-reserved=""
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341398    2581 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341401    2581 flags.go:64] FLAG: --register-node="true"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341404    2581 flags.go:64] FLAG: --register-with-taints=""
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341408    2581 flags.go:64] FLAG: --registry-burst="10"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341411    2581 flags.go:64] FLAG: --registry-qps="5"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341414    2581 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341416    2581 flags.go:64] FLAG: --reserved-memory=""
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341420    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341423    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341426    2581 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341429    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 17:59:52.348153 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341432    2581 flags.go:64] FLAG: --runonce="false"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341435    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341439    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341442    2581 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341444    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341447    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341450    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341453    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341456    2581 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341459    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341462    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341465    2581 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341468    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341472    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341476    2581 flags.go:64] FLAG: --system-cgroups=""
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341479    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341484    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341487    2581 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341490    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341494    2581 flags.go:64] FLAG: --tls-min-version=""
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341497    2581 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341500    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341503    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341506    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341509    2581 flags.go:64] FLAG: --v="2"
Aug 04 17:59:52.348781 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341512    2581 flags.go:64] FLAG: --version="false"
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341516    2581 flags.go:64] FLAG: --vmodule=""
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341520    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.341523    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341655    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341659    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341662    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341665    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341668    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341671    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341674    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341676    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341679    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341682    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341684    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341688    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341692    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341695    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341698    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:52.349418 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341701    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341705    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341708    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341711    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341714    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341717    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341719    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341722    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341725    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341727    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341730    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341733    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341736    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341738    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341741    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341744    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341747    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341749    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341752    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341755    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:52.349903 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341760    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341763    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341766    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341768    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341771    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341774    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341776    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341779    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341782    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341784    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341787    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341790    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341792    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341796    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341799    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341802    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341805    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341808    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341810    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341813    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:52.350394 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341815    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341818    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341821    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341823    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341826    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341829    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341832    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341834    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341837    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341840    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341842    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341845    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341847    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341850    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341853    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341856    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341858    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341861    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341863    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341866    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:52.350879 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341868    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341871    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341873    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341876    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341879    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341882    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341885    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341889    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341892    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341895    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341897    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341900    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341902    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341905    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341909    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341912    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341915    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341918    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341921    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:52.351362 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341923    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341926    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341929    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341932    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.341935    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.342485    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.349035    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.349050    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.349066    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.349071    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349129    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349134    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349138    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349141    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:52.351830 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349144    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349147    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349150    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349152    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349155    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349158    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349160    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349163    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349167    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349170    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349173    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349175    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349178    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349180    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349183    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349185    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349188    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349190    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349193    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349195    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:52.352215 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349197    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349200    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349202    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349205    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349207    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349211    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349214    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349216    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349219    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349222    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349224    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349227    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349230    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349232    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349235    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349239    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349242    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349245    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349248    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:52.352791 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349250    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349253    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349256    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349259    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349262    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349265    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349267    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349270    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349272    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349275    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349277    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349279    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349282    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349285    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349288    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349290    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349293    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349295    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349298    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349301    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:52.353238 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349304    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349306    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349309    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349312    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349314    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349317    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349319    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349321    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349324    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349327    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349329    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349331    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349334    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349336    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349340    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349351    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349354    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349357    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349360    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:52.353747 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349362    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349366    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349370    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349372    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349375    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349378    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349381    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349383    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349386    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349388    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349391    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349394    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349397    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349399    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349402    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349405    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:52.354202 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349407    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.349412    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349522    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349526    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349529    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349532    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349535    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349537    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349540    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349543    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349545    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349548    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349550    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349553    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349555    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:52.354596 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349558    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349560    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349563    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349565    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349568    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349570    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349573    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349575    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349578    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349581    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349583    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349586    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349589    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349591    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349594    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349597    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349616    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349619    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349623    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:52.354995 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349626    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349628    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349631    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349634    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349636    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349639    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349642    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349644    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349647    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349649    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349652    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349654    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349657    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349660    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349663    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349665    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349668    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349670    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349673    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349675    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:52.355557 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349678    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349680    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349683    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349686    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349689    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349691    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349694    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349696    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349699    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349701    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349704    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349707    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349710    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349713    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349716    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349719    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349721    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349724    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349726    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:52.356103 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349729    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349732    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349734    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349737    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349739    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349742    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349745    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349748    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349750    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349754    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349758    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349761    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349764    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349766    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349770    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349775    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349777    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349780    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349783    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:52.356914 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349786    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349788    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349791    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349793    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349796    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349798    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349801    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349803    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:52.349806    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.349811    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.349908    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.351925    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.353247    2581 server.go:1019] "Starting client certificate rotation"
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.353345    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 17:59:52.357614 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.353373    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 17:59:52.357978 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.355470    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 17:59:52.359658 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.359633    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 17:59:52.367911 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.367894    2581 log.go:25] "Validated CRI v1 runtime API"
Aug 04 17:59:52.370076 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.370063    2581 log.go:25] "Validated CRI v1 image API"
Aug 04 17:59:52.371613 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.371576    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 17:59:52.374376 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.374356    2581 fs.go:135] Filesystem UUIDs: map[29274e20-b00c-45a6-ab9f-5aaf0fa2e946:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c0d8457a-ea71-4cb5-a8dd-d16b5b9cba49:/dev/nvme0n1p4]
Aug 04 17:59:52.374448 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.374376    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 17:59:52.379948 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.379036    2581 manager.go:216] Machine: {Timestamp:2026-08-04 17:59:52.378540931 +0000 UTC m=+0.312610821 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3117483 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eede45cb3659353d675d66f3173df SystemUUID:ec2eede4-5cb3-6593-53d6-75d66f3173df BootID:383327af-69e1-4848-a691-894aed4c96ef Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:29:41:2c:fc:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:29:41:2c:fc:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:a2:4d:43:00:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 17:59:52.379948 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.379940    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 17:59:52.380113 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.380059    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 17:59:52.381455 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.381438    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 17:59:52.382381 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.382353    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 17:59:52.382581 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.382387    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-201.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 17:59:52.383198 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.383186    2581 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 17:59:52.383236 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.383202    2581 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 17:59:52.383236 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.383209    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 17:59:52.383678 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.383668    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 17:59:52.384450 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.384441    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 17:59:52.387762 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.387750    2581 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 17:59:52.387888 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.387878    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 17:59:52.387931 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.387910    2581 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 17:59:52.387931 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.387924    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 17:59:52.388020 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.387940    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 17:59:52.388020 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.387950    2581 kubelet.go:398] "Adding apiserver pod source"
Aug 04 17:59:52.388020 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.387967    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 17:59:52.388358 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.388346    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 17:59:52.388409 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.388364    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 17:59:52.391337 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.391319    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 17:59:52.392760 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.392746    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 17:59:52.392816 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.392776    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 17:59:52.394201 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394183    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 17:59:52.394262 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394211    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 17:59:52.394262 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394222    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 17:59:52.394262 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394231    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 17:59:52.394262 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394244    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 17:59:52.394262 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394253    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 17:59:52.394485 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394428    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 17:59:52.394485 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394449    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 17:59:52.394485 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394464    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 17:59:52.394485 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394475    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 17:59:52.394725 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394489    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 17:59:52.394725 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.394499    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 17:59:52.395742 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.395724    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 17:59:52.395742 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.395743    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 17:59:52.397089 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.397067    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 17:59:52.397164 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.397070    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 17:59:52.399748 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.399732    2581 server.go:1268] "Started kubelet"
Aug 04 17:59:52.399823 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.399784    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 17:59:52.399915 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.399868    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 17:59:52.399967 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.399957    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 17:59:52.400207 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.400105    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-201.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 17:59:52.400634 ip-10-0-139-201 systemd[1]: Started Kubernetes Kubelet.
Aug 04 17:59:52.401851 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.401837    2581 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 17:59:52.404102 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.404086    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 17:59:52.406408 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.405550    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-201.ec2.internal.18c8ac059f4ad90f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-201.ec2.internal,UID:ip-10-0-139-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-201.ec2.internal,},FirstTimestamp:2026-08-04 17:59:52.399698191 +0000 UTC m=+0.333768081,LastTimestamp:2026-08-04 17:59:52.399698191 +0000 UTC m=+0.333768081,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-201.ec2.internal,}"
Aug 04 17:59:52.406584 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.406568    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 17:59:52.409303 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.409288    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 17:59:52.409380 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.409301    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 17:59:52.409459 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.409440    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:52.410148 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410115    2581 factory.go:55] Registering systemd factory
Aug 04 17:59:52.410148 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410132    2581 factory.go:223] Registration of the systemd container factory successfully
Aug 04 17:59:52.410294 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410187    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 17:59:52.410294 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410200    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 17:59:52.410294 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410202    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 17:59:52.410408 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410308    2581 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 17:59:52.410408 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410315    2581 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 17:59:52.410408 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410339    2581 factory.go:153] Registering CRI-O factory
Aug 04 17:59:52.410408 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410354    2581 factory.go:223] Registration of the crio container factory successfully
Aug 04 17:59:52.410408 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410410    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 17:59:52.410649 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410436    2581 factory.go:103] Registering Raw factory
Aug 04 17:59:52.410649 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410448    2581 manager.go:1196] Started watching for new ooms in manager
Aug 04 17:59:52.410907 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.410894    2581 manager.go:349] Starting recovery of all containers
Aug 04 17:59:52.411316 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.411289    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 17:59:52.411433 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.411413    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 17:59:52.420110 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.420095    2581 manager.go:354] Recovery completed
Aug 04 17:59:52.424970 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.424951    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xlvc8"
Aug 04 17:59:52.425469 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.425457    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:52.427227 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.427212    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:52.427279 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.427239    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:52.427279 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.427249    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:52.427761 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.427743    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 17:59:52.427761 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.427759    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 17:59:52.427893 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.427778    2581 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 17:59:52.427893 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.427758    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:52.428507 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.428494    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:52.428552 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.428518    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:52.428552 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.428527    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:52.428967 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.428954    2581 policy_none.go:49] "None policy: Start"
Aug 04 17:59:52.428967 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.428969    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 17:59:52.429068 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.428979    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 17:59:52.429915 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.429903    2581 policy_none.go:47] "Start"
Aug 04 17:59:52.435459 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.435295    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-201.ec2.internal.18c8ac05a0eee7c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-201.ec2.internal,UID:ip-10-0-139-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-201.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-201.ec2.internal,},FirstTimestamp:2026-08-04 17:59:52.427227078 +0000 UTC m=+0.361296970,LastTimestamp:2026-08-04 17:59:52.427227078 +0000 UTC m=+0.361296970,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-201.ec2.internal,}"
Aug 04 17:59:52.436261 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.436246    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xlvc8"
Aug 04 17:59:52.437895 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.437837    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-201.ec2.internal.18c8ac05a0ef2a06  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-201.ec2.internal,UID:ip-10-0-139-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-201.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-201.ec2.internal,},FirstTimestamp:2026-08-04 17:59:52.427244038 +0000 UTC m=+0.361313928,LastTimestamp:2026-08-04 17:59:52.427244038 +0000 UTC m=+0.361313928,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-201.ec2.internal,}"
Aug 04 17:59:52.474166 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.474151    2581 manager.go:337] "Starting Device Plugin manager"
Aug 04 17:59:52.474259 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.474185    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 17:59:52.474259 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.474198    2581 server.go:87] "Starting device plugin registration server"
Aug 04 17:59:52.474472 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.474459    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 17:59:52.474566 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.474474    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 17:59:52.474633 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.474593    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 17:59:52.474699 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.474683    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 17:59:52.474735 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.474701    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 17:59:52.475133 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.475116    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 17:59:52.475192 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.475158    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:52.538564 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.538524    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 17:59:52.539731 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.539715    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 17:59:52.539801 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.539755    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 17:59:52.539801 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.539778    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 17:59:52.539881 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.539818    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 17:59:52.547021 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.546997    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:52.575658 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.575639    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:52.576550 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.576532    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:52.576629 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.576568    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:52.576629 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.576585    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:52.576629 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.576616    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.583771 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.583725    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.583771 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.583749    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-201.ec2.internal\": node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:52.600735 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.600714    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:52.640509 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.640482    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"]
Aug 04 17:59:52.640652 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.640549    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:52.642201 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.642184    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:52.642292 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.642213    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:52.642292 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.642223    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:52.643514 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.643502    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:52.643666 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.643650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.643725 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.643683    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:52.644260 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.644241    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:52.644343 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.644275    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:52.644343 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.644248    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:52.644343 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.644315    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:52.644343 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.644328    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:52.644343 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.644286    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:52.645596 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.645582    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.645654 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.645606    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:52.646382 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.646363    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:52.646443 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.646397    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:52.646443 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.646416    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:52.669411 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.669384    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.673991 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.673974    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.701012 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.700989    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:52.712474 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.712453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd653a884275f59e7e79ba36a5c01dd0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"bd653a884275f59e7e79ba36a5c01dd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.712553 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.712480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd653a884275f59e7e79ba36a5c01dd0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"bd653a884275f59e7e79ba36a5c01dd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.712553 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.712511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ced7042c6bb32bfc2b5d6d18b0919b8f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"ced7042c6bb32bfc2b5d6d18b0919b8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.802094 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.802032    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:52.813269 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.813236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd653a884275f59e7e79ba36a5c01dd0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"bd653a884275f59e7e79ba36a5c01dd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.813391 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.813276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd653a884275f59e7e79ba36a5c01dd0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"bd653a884275f59e7e79ba36a5c01dd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.813391 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.813302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ced7042c6bb32bfc2b5d6d18b0919b8f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"ced7042c6bb32bfc2b5d6d18b0919b8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.813391 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.813360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd653a884275f59e7e79ba36a5c01dd0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"bd653a884275f59e7e79ba36a5c01dd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.813513 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.813387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd653a884275f59e7e79ba36a5c01dd0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"bd653a884275f59e7e79ba36a5c01dd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.813513 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.813361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ced7042c6bb32bfc2b5d6d18b0919b8f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"ced7042c6bb32bfc2b5d6d18b0919b8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.902876 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:52.902831    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:52.976985 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.976951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:52.977102 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:52.976951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:53.003947 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:53.003911    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:53.104237 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:53.104190    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:53.204878 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:53.204801    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:53.305889 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:53.305848    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:53.331770 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.331742    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:53.352577 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.352549    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 17:59:53.352991 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.352728    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 17:59:53.352991 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.352726    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 17:59:53.406563 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:53.406540    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:53.439728 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.439686    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:54:52 +0000 UTC" deadline="2028-03-31 05:06:40.198001847 +0000 UTC"
Aug 04 17:59:53.439728 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.439723    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14507h6m46.758282228s"
Aug 04 17:59:53.501282 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:53.501243    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd653a884275f59e7e79ba36a5c01dd0.slice/crio-8e5f53f0116d854aa7476ddb66b78b424862f36cd8013e833fe250c988d2cf9d WatchSource:0}: Error finding container 8e5f53f0116d854aa7476ddb66b78b424862f36cd8013e833fe250c988d2cf9d: Status 404 returned error can't find the container with id 8e5f53f0116d854aa7476ddb66b78b424862f36cd8013e833fe250c988d2cf9d
Aug 04 17:59:53.507818 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:53.506789    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:53.508599 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.508578    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 17:59:53.509241 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:53.509209    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced7042c6bb32bfc2b5d6d18b0919b8f.slice/crio-3841a9bce577646a72ca2209125a9eba9f8f587ec3b8c8344f7be9b1fede7743 WatchSource:0}: Error finding container 3841a9bce577646a72ca2209125a9eba9f8f587ec3b8c8344f7be9b1fede7743: Status 404 returned error can't find the container with id 3841a9bce577646a72ca2209125a9eba9f8f587ec3b8c8344f7be9b1fede7743
Aug 04 17:59:53.543533 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.543466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" event={"ID":"ced7042c6bb32bfc2b5d6d18b0919b8f","Type":"ContainerStarted","Data":"3841a9bce577646a72ca2209125a9eba9f8f587ec3b8c8344f7be9b1fede7743"}
Aug 04 17:59:53.544585 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.544558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"bd653a884275f59e7e79ba36a5c01dd0","Type":"ContainerStarted","Data":"8e5f53f0116d854aa7476ddb66b78b424862f36cd8013e833fe250c988d2cf9d"}
Aug 04 17:59:53.604274 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.604240    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:53.605695 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.605676    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:53.605772 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.605711    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:53.605772 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.605723    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:53.607791 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:53.607775    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found"
Aug 04 17:59:53.625561 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.625540    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:53.631771 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.631750    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:53.710499 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.710472    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:53.724265 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.724240    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 17:59:53.725239 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.725225    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"
Aug 04 17:59:53.735507 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:53.735485    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 17:59:54.389693 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.389661    2581 apiserver.go:52] "Watching apiserver"
Aug 04 17:59:54.400336 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.400309    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 17:59:54.400749 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.400724    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp","openshift-multus/multus-additional-cni-plugins-lh9j2","openshift-multus/multus-srl6l","openshift-multus/network-metrics-daemon-x6dkv","openshift-network-diagnostics/network-check-target-rbnmz","openshift-ovn-kubernetes/ovnkube-node-ncgkr","kube-system/global-pull-secret-syncer-glgg2","kube-system/konnectivity-agent-9rkkz","kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal","openshift-cluster-node-tuning-operator/tuned-sz4dp","openshift-dns/node-resolver-rl4zg","openshift-image-registry/node-ca-mm2xh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal","openshift-network-operator/iptables-alerter-9vhwg"]
Aug 04 17:59:54.402353 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.402314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:54.402474 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.402399    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 17:59:54.403595 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.403573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.404844 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.404827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.406228 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.406205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:54.406326 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.406274    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 17:59:54.407618 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.407549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 17:59:54.407737 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.407633    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 17:59:54.409042 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.408986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.409539 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.409514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9rmkq\""
Aug 04 17:59:54.409692 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.409675    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 17:59:54.410080 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.410061    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 17:59:54.410148 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.410116    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 17:59:54.410344 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.410267    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 17:59:54.410344 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.410343    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 17:59:54.410569 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.410412    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 17:59:54.410678 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.410582    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.410678 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.410607    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 17:59:54.410962 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.410874    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jc6bm\""
Aug 04 17:59:54.412291 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.411892    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 17:59:54.412291 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.412044    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 17:59:54.412291 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.412250    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kpct\""
Aug 04 17:59:54.412686 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.412597    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 17:59:54.412931 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.412873    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 17:59:54.417719 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.416471    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 17:59:54.417719 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.417459    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 17:59:54.417719 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.417521    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 17:59:54.419336 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.418043    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 17:59:54.419336 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.418104    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f4srf\""
Aug 04 17:59:54.419336 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.418137    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 17:59:54.419336 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.418265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.419336 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.416475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 17:59:54.420607 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.419727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.422896 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.422695    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.424137 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8sp\" (UniqueName: \"kubernetes.io/projected/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-kube-api-access-qx8sp\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:54.424236 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-etc-openvswitch\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424236 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-run-openvswitch\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424236 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4ba9272-9ed2-418b-91ce-7f116ee67171-ovn-node-metrics-cert\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424403 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5fl7\" (UniqueName: \"kubernetes.io/projected/1e8bdc52-e2e6-4679-8ff8-de9ba96c3770-kube-api-access-f5fl7\") pod \"node-resolver-rl4zg\" (UID: \"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770\") " pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.424403 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-run-netns\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.424403 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-conf-dir\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.424403 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.424403 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:54.424403 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-node-log\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-cni-bin\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-os-release\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-daemon-config\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-systemd-units\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-run-systemd\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-log-socket\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lrqc\" (UniqueName: \"kubernetes.io/projected/a4ba9272-9ed2-418b-91ce-7f116ee67171-kube-api-access-8lrqc\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71e1a397-17f5-4aba-98d7-7a8a0a7be2de-iptables-alerter-script\") pod \"iptables-alerter-9vhwg\" (UID: \"71e1a397-17f5-4aba-98d7-7a8a0a7be2de\") " pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e8bdc52-e2e6-4679-8ff8-de9ba96c3770-hosts-file\") pod \"node-resolver-rl4zg\" (UID: \"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770\") " pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8bdc52-e2e6-4679-8ff8-de9ba96c3770-tmp-dir\") pod \"node-resolver-rl4zg\" (UID: \"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770\") " pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.424721 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-run-ovn\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a4ba9272-9ed2-418b-91ce-7f116ee67171-env-overrides\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-system-cni-dir\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-cni-dir\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-socket-dir-parent\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-var-lib-cni-bin\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-var-lib-cni-multus\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/082c2f9a-47d7-4425-90dc-55ee974b6c58-dbus\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-var-lib-openvswitch\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a4ba9272-9ed2-418b-91ce-7f116ee67171-ovnkube-config\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71e1a397-17f5-4aba-98d7-7a8a0a7be2de-host-slash\") pod \"iptables-alerter-9vhwg\" (UID: \"71e1a397-17f5-4aba-98d7-7a8a0a7be2de\") " pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.424999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-var-lib-kubelet\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-hostroot\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtv9\" (UniqueName: \"kubernetes.io/projected/f067a520-b4c3-47f8-b5af-9eb7ea701266-kube-api-access-jrtv9\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-kubelet\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.425441 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-run-netns\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-cni-netd\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-cnibin\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425087    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f067a520-b4c3-47f8-b5af-9eb7ea701266-cni-binary-copy\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-run-k8s-cni-cncf-io\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-etc-kubernetes\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-run-ovn-kubernetes\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smc4l\" (UniqueName: \"kubernetes.io/projected/71e1a397-17f5-4aba-98d7-7a8a0a7be2de-kube-api-access-smc4l\") pod \"iptables-alerter-9vhwg\" (UID: \"71e1a397-17f5-4aba-98d7-7a8a0a7be2de\") " pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/082c2f9a-47d7-4425-90dc-55ee974b6c58-kubelet-config\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-slash\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a4ba9272-9ed2-418b-91ce-7f116ee67171-ovnkube-script-lib\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-run-multus-certs\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6lmdn\""
Aug 04 17:59:54.426182 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.425906    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 17:59:54.426845 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.426211    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 17:59:54.426845 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.426235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n48gm\""
Aug 04 17:59:54.426845 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.426472    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 17:59:54.426845 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.426525    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 17:59:54.427037 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.426898    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 17:59:54.427150 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.427125    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 17:59:54.427223 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.427170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-npbs9\""
Aug 04 17:59:54.427555 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.427377    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 17:59:54.427627 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.427559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 17:59:54.427627 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.427600    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nccl5\""
Aug 04 17:59:54.428186 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.428058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pjmtd\""
Aug 04 17:59:54.428699 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.428680    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 17:59:54.428925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.428909    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 17:59:54.429016 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.428999    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 17:59:54.433529 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.433504    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 17:59:54.450079 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.450054    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-thcsx"
Aug 04 17:59:54.457322 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.457300    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-thcsx"
Aug 04 17:59:54.501357 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.501329    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:54.512198 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.512166    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 17:59:54.526435 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.526594 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.526594 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-modprobe-d\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.526594 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:54.526594 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-kubelet\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.526917 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-run-netns\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.526917 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-kubelet\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.526917 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-run-netns\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.526917 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-cni-netd\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.526917 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.526842    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:54.526917 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-cnibin\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.526917 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-cni-netd\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-run-k8s-cni-cncf-io\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.526977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-systemd\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-run-k8s-cni-cncf-io\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-lib-modules\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-run-ovn-kubernetes\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-kubernetes\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-var-lib-kubelet\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.527133    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs podName:a8e5dc99-1596-4919-84a4-f8d3bc43ca2d nodeName:}" failed. No retries permitted until 2026-08-04 17:59:55.027097215 +0000 UTC m=+2.961167131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs") pod "network-metrics-daemon-x6dkv" (UID: "a8e5dc99-1596-4919-84a4-f8d3bc43ca2d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-run-ovn-kubernetes\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.527197 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-cnibin\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmz5c\" (UniqueName: \"kubernetes.io/projected/24deb436-b82d-434c-b34b-f85e19cc147f-kube-api-access-kmz5c\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-registration-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-sys-fs\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/082c2f9a-47d7-4425-90dc-55ee974b6c58-kubelet-config\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-slash\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-run-multus-certs\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-run-multus-certs\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-system-cni-dir\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-run\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-slash\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/082c2f9a-47d7-4425-90dc-55ee974b6c58-kubelet-config\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24deb436-b82d-434c-b34b-f85e19cc147f-tmp\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.527641 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7sg\" (UniqueName: \"kubernetes.io/projected/1379cd7a-a7b5-44da-9387-490611d058ef-kube-api-access-jv7sg\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4ba9272-9ed2-418b-91ce-7f116ee67171-ovn-node-metrics-cert\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5fl7\" (UniqueName: \"kubernetes.io/projected/1e8bdc52-e2e6-4679-8ff8-de9ba96c3770-kube-api-access-f5fl7\") pod \"node-resolver-rl4zg\" (UID: \"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770\") " pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-run-netns\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-conf-dir\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f50134d-2df8-49c9-93fc-1337b4b52b3b-agent-certs\") pod \"konnectivity-agent-9rkkz\" (UID: \"1f50134d-2df8-49c9-93fc-1337b4b52b3b\") " pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-cni-bin\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.527969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f50134d-2df8-49c9-93fc-1337b4b52b3b-konnectivity-ca\") pod \"konnectivity-agent-9rkkz\" (UID: \"1f50134d-2df8-49c9-93fc-1337b4b52b3b\") " pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-systemd-units\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-run-systemd\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lrqc\" (UniqueName: \"kubernetes.io/projected/a4ba9272-9ed2-418b-91ce-7f116ee67171-kube-api-access-8lrqc\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71e1a397-17f5-4aba-98d7-7a8a0a7be2de-iptables-alerter-script\") pod \"iptables-alerter-9vhwg\" (UID: \"71e1a397-17f5-4aba-98d7-7a8a0a7be2de\") " pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e8bdc52-e2e6-4679-8ff8-de9ba96c3770-hosts-file\") pod \"node-resolver-rl4zg\" (UID: \"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770\") " pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.528197    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528183    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8bdc52-e2e6-4679-8ff8-de9ba96c3770-tmp-dir\") pod \"node-resolver-rl4zg\" (UID: \"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770\") " pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.528319 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.528253    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret podName:082c2f9a-47d7-4425-90dc-55ee974b6c58 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:55.028237801 +0000 UTC m=+2.962307683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret") pod "global-pull-secret-syncer-glgg2" (UID: "082c2f9a-47d7-4425-90dc-55ee974b6c58") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-conf-dir\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-sysconfig\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8dx5\" (UniqueName: \"kubernetes.io/projected/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-kube-api-access-n8dx5\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caccac04-758f-4f30-91d8-d2c828ab88d7-host\") pod \"node-ca-mm2xh\" (UID: \"caccac04-758f-4f30-91d8-d2c828ab88d7\") " pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/082c2f9a-47d7-4425-90dc-55ee974b6c58-dbus\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71e1a397-17f5-4aba-98d7-7a8a0a7be2de-host-slash\") pod \"iptables-alerter-9vhwg\" (UID: \"71e1a397-17f5-4aba-98d7-7a8a0a7be2de\") " pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-hostroot\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f067a520-b4c3-47f8-b5af-9eb7ea701266-cni-binary-copy\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-etc-kubernetes\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smc4l\" (UniqueName: \"kubernetes.io/projected/71e1a397-17f5-4aba-98d7-7a8a0a7be2de-kube-api-access-smc4l\") pod \"iptables-alerter-9vhwg\" (UID: \"71e1a397-17f5-4aba-98d7-7a8a0a7be2de\") " pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8bdc52-e2e6-4679-8ff8-de9ba96c3770-tmp-dir\") pod \"node-resolver-rl4zg\" (UID: \"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770\") " pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-sys\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-cni-bin\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.529164 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/caccac04-758f-4f30-91d8-d2c828ab88d7-serviceca\") pod \"node-ca-mm2xh\" (UID: \"caccac04-758f-4f30-91d8-d2c828ab88d7\") " pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-run-netns\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a4ba9272-9ed2-418b-91ce-7f116ee67171-ovnkube-script-lib\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-socket-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8sp\" (UniqueName: \"kubernetes.io/projected/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-kube-api-access-qx8sp\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-etc-openvswitch\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-run-openvswitch\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.528983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-sysctl-d\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-node-log\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-os-release\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-daemon-config\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-run-systemd\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-cnibin\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-os-release\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-systemd-units\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-sysctl-conf\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.530032 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-device-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-log-socket\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-socket-dir-parent\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71e1a397-17f5-4aba-98d7-7a8a0a7be2de-host-slash\") pod \"iptables-alerter-9vhwg\" (UID: \"71e1a397-17f5-4aba-98d7-7a8a0a7be2de\") " pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-var-lib-cni-multus\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ltn6\" (UniqueName: \"kubernetes.io/projected/caccac04-758f-4f30-91d8-d2c828ab88d7-kube-api-access-6ltn6\") pod \"node-ca-mm2xh\" (UID: \"caccac04-758f-4f30-91d8-d2c828ab88d7\") " pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-run-ovn\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-var-lib-cni-multus\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a4ba9272-9ed2-418b-91ce-7f116ee67171-env-overrides\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-system-cni-dir\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a4ba9272-9ed2-418b-91ce-7f116ee67171-ovnkube-script-lib\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-etc-kubernetes\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e8bdc52-e2e6-4679-8ff8-de9ba96c3770-hosts-file\") pod \"node-resolver-rl4zg\" (UID: \"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770\") " pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a4ba9272-9ed2-418b-91ce-7f116ee67171-env-overrides\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530016    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-log-socket\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.530826 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.529305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/082c2f9a-47d7-4425-90dc-55ee974b6c58-dbus\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-socket-dir-parent\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-cni-dir\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-var-lib-cni-bin\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-cni-dir\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-host\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24deb436-b82d-434c-b34b-f85e19cc147f-etc-tuned\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-system-cni-dir\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-var-lib-openvswitch\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-hostroot\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a4ba9272-9ed2-418b-91ce-7f116ee67171-ovnkube-config\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530288    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-etc-openvswitch\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f067a520-b4c3-47f8-b5af-9eb7ea701266-cni-binary-copy\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-run-openvswitch\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f067a520-b4c3-47f8-b5af-9eb7ea701266-multus-daemon-config\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-os-release\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-run-ovn\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-var-lib-cni-bin\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.531645 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-var-lib-kubelet\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.532382 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtv9\" (UniqueName: \"kubernetes.io/projected/f067a520-b4c3-47f8-b5af-9eb7ea701266-kube-api-access-jrtv9\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.532382 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-var-lib-openvswitch\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.532382 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71e1a397-17f5-4aba-98d7-7a8a0a7be2de-iptables-alerter-script\") pod \"iptables-alerter-9vhwg\" (UID: \"71e1a397-17f5-4aba-98d7-7a8a0a7be2de\") " pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.532382 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 17:59:54.532382 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f067a520-b4c3-47f8-b5af-9eb7ea701266-host-var-lib-kubelet\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.532382 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a4ba9272-9ed2-418b-91ce-7f116ee67171-node-log\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.532382 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.530748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a4ba9272-9ed2-418b-91ce-7f116ee67171-ovnkube-config\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.533041 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.533015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4ba9272-9ed2-418b-91ce-7f116ee67171-ovn-node-metrics-cert\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.538915 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.538832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5fl7\" (UniqueName: \"kubernetes.io/projected/1e8bdc52-e2e6-4679-8ff8-de9ba96c3770-kube-api-access-f5fl7\") pod \"node-resolver-rl4zg\" (UID: \"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770\") " pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.544576 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.543849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lrqc\" (UniqueName: \"kubernetes.io/projected/a4ba9272-9ed2-418b-91ce-7f116ee67171-kube-api-access-8lrqc\") pod \"ovnkube-node-ncgkr\" (UID: \"a4ba9272-9ed2-418b-91ce-7f116ee67171\") " pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.545515 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.545238    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:54.545515 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.545274    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:54.545515 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.545290    2581 projected.go:196] Error preparing data for projected volume kube-api-access-tnb79 for pod openshift-network-diagnostics/network-check-target-rbnmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:54.545515 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:54.545393    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79 podName:87ce351b-62f5-44a3-89a7-ed4a18221c8a nodeName:}" failed. No retries permitted until 2026-08-04 17:59:55.045371772 +0000 UTC m=+2.979441678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tnb79" (UniqueName: "kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79") pod "network-check-target-rbnmz" (UID: "87ce351b-62f5-44a3-89a7-ed4a18221c8a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:54.548623 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.548596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtv9\" (UniqueName: \"kubernetes.io/projected/f067a520-b4c3-47f8-b5af-9eb7ea701266-kube-api-access-jrtv9\") pod \"multus-srl6l\" (UID: \"f067a520-b4c3-47f8-b5af-9eb7ea701266\") " pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.548809 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.548786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8sp\" (UniqueName: \"kubernetes.io/projected/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-kube-api-access-qx8sp\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:54.549485 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.549445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smc4l\" (UniqueName: \"kubernetes.io/projected/71e1a397-17f5-4aba-98d7-7a8a0a7be2de-kube-api-access-smc4l\") pod \"iptables-alerter-9vhwg\" (UID: \"71e1a397-17f5-4aba-98d7-7a8a0a7be2de\") " pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.631492 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-host\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.631690 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24deb436-b82d-434c-b34b-f85e19cc147f-etc-tuned\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.631690 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-host\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.631690 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.631690 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.631925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-modprobe-d\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.631925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-systemd\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.631925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-lib-modules\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.631925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-kubernetes\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.631925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-var-lib-kubelet\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.631925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmz5c\" (UniqueName: \"kubernetes.io/projected/24deb436-b82d-434c-b34b-f85e19cc147f-kube-api-access-kmz5c\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.631925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-registration-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.631925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-sys-fs\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.631925 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-modprobe-d\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-system-cni-dir\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-run\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24deb436-b82d-434c-b34b-f85e19cc147f-tmp\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.631987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-var-lib-kubelet\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7sg\" (UniqueName: \"kubernetes.io/projected/1379cd7a-a7b5-44da-9387-490611d058ef-kube-api-access-jv7sg\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-systemd\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f50134d-2df8-49c9-93fc-1337b4b52b3b-agent-certs\") pod \"konnectivity-agent-9rkkz\" (UID: \"1f50134d-2df8-49c9-93fc-1337b4b52b3b\") " pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f50134d-2df8-49c9-93fc-1337b4b52b3b-konnectivity-ca\") pod \"konnectivity-agent-9rkkz\" (UID: \"1f50134d-2df8-49c9-93fc-1337b4b52b3b\") " pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-sysconfig\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dx5\" (UniqueName: \"kubernetes.io/projected/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-kube-api-access-n8dx5\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caccac04-758f-4f30-91d8-d2c828ab88d7-host\") pod \"node-ca-mm2xh\" (UID: \"caccac04-758f-4f30-91d8-d2c828ab88d7\") " pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-sys\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/caccac04-758f-4f30-91d8-d2c828ab88d7-serviceca\") pod \"node-ca-mm2xh\" (UID: \"caccac04-758f-4f30-91d8-d2c828ab88d7\") " pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.632373 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-system-cni-dir\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-socket-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-sysctl-d\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-lib-modules\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-cnibin\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-os-release\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caccac04-758f-4f30-91d8-d2c828ab88d7-host\") pod \"node-ca-mm2xh\" (UID: \"caccac04-758f-4f30-91d8-d2c828ab88d7\") " pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-kubernetes\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-sysctl-conf\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-device-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ltn6\" (UniqueName: \"kubernetes.io/projected/caccac04-758f-4f30-91d8-d2c828ab88d7-kube-api-access-6ltn6\") pod \"node-ca-mm2xh\" (UID: \"caccac04-758f-4f30-91d8-d2c828ab88d7\") " pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-run\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-sys-fs\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.633271 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-registration-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/caccac04-758f-4f30-91d8-d2c828ab88d7-serviceca\") pod \"node-ca-mm2xh\" (UID: \"caccac04-758f-4f30-91d8-d2c828ab88d7\") " pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632855    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-socket-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-sysctl-d\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-cnibin\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f50134d-2df8-49c9-93fc-1337b4b52b3b-konnectivity-ca\") pod \"konnectivity-agent-9rkkz\" (UID: \"1f50134d-2df8-49c9-93fc-1337b4b52b3b\") " pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.632592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-sys\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.633052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1379cd7a-a7b5-44da-9387-490611d058ef-device-dir\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.633057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-os-release\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.633100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-sysconfig\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.633234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.633880 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.633346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24deb436-b82d-434c-b34b-f85e19cc147f-etc-sysctl-conf\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.634344 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.634190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24deb436-b82d-434c-b34b-f85e19cc147f-etc-tuned\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.635178 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.635155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24deb436-b82d-434c-b34b-f85e19cc147f-tmp\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.635569 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.635544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f50134d-2df8-49c9-93fc-1337b4b52b3b-agent-certs\") pod \"konnectivity-agent-9rkkz\" (UID: \"1f50134d-2df8-49c9-93fc-1337b4b52b3b\") " pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 17:59:54.650363 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.650301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dx5\" (UniqueName: \"kubernetes.io/projected/d7e66ed3-d135-4ce6-af93-b463a55a2b6f-kube-api-access-n8dx5\") pod \"multus-additional-cni-plugins-lh9j2\" (UID: \"d7e66ed3-d135-4ce6-af93-b463a55a2b6f\") " pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.651467 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.651445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ltn6\" (UniqueName: \"kubernetes.io/projected/caccac04-758f-4f30-91d8-d2c828ab88d7-kube-api-access-6ltn6\") pod \"node-ca-mm2xh\" (UID: \"caccac04-758f-4f30-91d8-d2c828ab88d7\") " pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:54.651553 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.651475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7sg\" (UniqueName: \"kubernetes.io/projected/1379cd7a-a7b5-44da-9387-490611d058ef-kube-api-access-jv7sg\") pod \"aws-ebs-csi-driver-node-vxblp\" (UID: \"1379cd7a-a7b5-44da-9387-490611d058ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.651553 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.651490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmz5c\" (UniqueName: \"kubernetes.io/projected/24deb436-b82d-434c-b34b-f85e19cc147f-kube-api-access-kmz5c\") pod \"tuned-sz4dp\" (UID: \"24deb436-b82d-434c-b34b-f85e19cc147f\") " pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.718907 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.718867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rl4zg"
Aug 04 17:59:54.729963 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.729937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-srl6l"
Aug 04 17:59:54.739227 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.739203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 17:59:54.748889 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.748868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9vhwg"
Aug 04 17:59:54.756660 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.756638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lh9j2"
Aug 04 17:59:54.765300 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.765279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 17:59:54.773945 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.773921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sz4dp"
Aug 04 17:59:54.782540 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.782520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp"
Aug 04 17:59:54.789302 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:54.789283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mm2xh"
Aug 04 17:59:55.035145 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:55.035056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:55.035322 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.035214    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:55.035322 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:55.035251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:55.035322 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.035284    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret podName:082c2f9a-47d7-4425-90dc-55ee974b6c58 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:56.035267981 +0000 UTC m=+3.969337860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret") pod "global-pull-secret-syncer-glgg2" (UID: "082c2f9a-47d7-4425-90dc-55ee974b6c58") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:55.035478 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.035362    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:55.035478 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.035411    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs podName:a8e5dc99-1596-4919-84a4-f8d3bc43ca2d nodeName:}" failed. No retries permitted until 2026-08-04 17:59:56.03539586 +0000 UTC m=+3.969465758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs") pod "network-metrics-daemon-x6dkv" (UID: "a8e5dc99-1596-4919-84a4-f8d3bc43ca2d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:55.136228 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:55.136187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 17:59:55.136425 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.136389    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:55.136425 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.136419    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:55.136529 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.136436    2581 projected.go:196] Error preparing data for projected volume kube-api-access-tnb79 for pod openshift-network-diagnostics/network-check-target-rbnmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:55.136529 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.136511    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79 podName:87ce351b-62f5-44a3-89a7-ed4a18221c8a nodeName:}" failed. No retries permitted until 2026-08-04 17:59:56.136490977 +0000 UTC m=+4.070560874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnb79" (UniqueName: "kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79") pod "network-check-target-rbnmz" (UID: "87ce351b-62f5-44a3-89a7-ed4a18221c8a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:55.460531 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:55.460491    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:54:54 +0000 UTC" deadline="2028-04-29 05:51:02.363389638 +0000 UTC"
Aug 04 17:59:55.460531 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:55.460524    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15203h51m6.902870242s"
Aug 04 17:59:55.540229 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:55.540182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:55.540229 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:55.540222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 17:59:55.540465 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:55.540181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:55.540465 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.540342    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 17:59:55.540465 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.540444    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 17:59:55.540622 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:55.540542    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 17:59:55.572388 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:55.572359    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e66ed3_d135_4ce6_af93_b463a55a2b6f.slice/crio-03b06869e782e11274451a710ae26880ca1d0f4577b6fe30dca095656fed7ea5 WatchSource:0}: Error finding container 03b06869e782e11274451a710ae26880ca1d0f4577b6fe30dca095656fed7ea5: Status 404 returned error can't find the container with id 03b06869e782e11274451a710ae26880ca1d0f4577b6fe30dca095656fed7ea5
Aug 04 17:59:55.574296 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:55.574256    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f50134d_2df8_49c9_93fc_1337b4b52b3b.slice/crio-3fa35865483c58ea8d152412f74949c131701c54a5a3a2fabf84b5abbe66eeb0 WatchSource:0}: Error finding container 3fa35865483c58ea8d152412f74949c131701c54a5a3a2fabf84b5abbe66eeb0: Status 404 returned error can't find the container with id 3fa35865483c58ea8d152412f74949c131701c54a5a3a2fabf84b5abbe66eeb0
Aug 04 17:59:55.575304 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:55.575279    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24deb436_b82d_434c_b34b_f85e19cc147f.slice/crio-37ebef7273ee88e90b97ead1a5da5bd553f9df816e058ec676d9ae0be5d8a616 WatchSource:0}: Error finding container 37ebef7273ee88e90b97ead1a5da5bd553f9df816e058ec676d9ae0be5d8a616: Status 404 returned error can't find the container with id 37ebef7273ee88e90b97ead1a5da5bd553f9df816e058ec676d9ae0be5d8a616
Aug 04 17:59:55.576331 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:55.576253    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf067a520_b4c3_47f8_b5af_9eb7ea701266.slice/crio-9c2211cc89a1753b01071e6faaa9919a45ada423228e235e1f91dd92b2e11f60 WatchSource:0}: Error finding container 9c2211cc89a1753b01071e6faaa9919a45ada423228e235e1f91dd92b2e11f60: Status 404 returned error can't find the container with id 9c2211cc89a1753b01071e6faaa9919a45ada423228e235e1f91dd92b2e11f60
Aug 04 17:59:55.579455 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:55.579423    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaccac04_758f_4f30_91d8_d2c828ab88d7.slice/crio-989998da73aed507021dc357f858f760d2c0e43bd896ea60673d783620309623 WatchSource:0}: Error finding container 989998da73aed507021dc357f858f760d2c0e43bd896ea60673d783620309623: Status 404 returned error can't find the container with id 989998da73aed507021dc357f858f760d2c0e43bd896ea60673d783620309623
Aug 04 17:59:55.579862 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:55.579838    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8bdc52_e2e6_4679_8ff8_de9ba96c3770.slice/crio-134b9802f6e6b7d415eff1767cd8435374ef37b318d2553f2d939fb41381b33e WatchSource:0}: Error finding container 134b9802f6e6b7d415eff1767cd8435374ef37b318d2553f2d939fb41381b33e: Status 404 returned error can't find the container with id 134b9802f6e6b7d415eff1767cd8435374ef37b318d2553f2d939fb41381b33e
Aug 04 17:59:55.585037 ip-10-0-139-201 kubenswrapper[2581]: W0804 17:59:55.584454    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ba9272_9ed2_418b_91ce_7f116ee67171.slice/crio-547fa1f28402e09689ae2247cdaa889aa68e05204a2227156e937927f37bd147 WatchSource:0}: Error finding container 547fa1f28402e09689ae2247cdaa889aa68e05204a2227156e937927f37bd147: Status 404 returned error can't find the container with id 547fa1f28402e09689ae2247cdaa889aa68e05204a2227156e937927f37bd147
Aug 04 17:59:56.044634 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.043771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:56.044634 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.043845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:56.044634 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:56.044005    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:56.044634 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:56.044072    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret podName:082c2f9a-47d7-4425-90dc-55ee974b6c58 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:58.044052406 +0000 UTC m=+5.978122287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret") pod "global-pull-secret-syncer-glgg2" (UID: "082c2f9a-47d7-4425-90dc-55ee974b6c58") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:56.044634 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:56.044495    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:56.044634 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:56.044550    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs podName:a8e5dc99-1596-4919-84a4-f8d3bc43ca2d nodeName:}" failed. No retries permitted until 2026-08-04 17:59:58.044533401 +0000 UTC m=+5.978603286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs") pod "network-metrics-daemon-x6dkv" (UID: "a8e5dc99-1596-4919-84a4-f8d3bc43ca2d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:56.144539 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.144493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 17:59:56.144747 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:56.144714    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:56.144817 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:56.144751    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:56.144817 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:56.144768    2581 projected.go:196] Error preparing data for projected volume kube-api-access-tnb79 for pod openshift-network-diagnostics/network-check-target-rbnmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:56.144931 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:56.144831    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79 podName:87ce351b-62f5-44a3-89a7-ed4a18221c8a nodeName:}" failed. No retries permitted until 2026-08-04 17:59:58.144810761 +0000 UTC m=+6.078880642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnb79" (UniqueName: "kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79") pod "network-check-target-rbnmz" (UID: "87ce351b-62f5-44a3-89a7-ed4a18221c8a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:56.356738 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.356359    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:56.460832 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.460746    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:54:54 +0000 UTC" deadline="2028-04-23 04:57:33.001287578 +0000 UTC"
Aug 04 17:59:56.460832 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.460808    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15058h57m36.540483668s"
Aug 04 17:59:56.562220 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.562179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rkkz" event={"ID":"1f50134d-2df8-49c9-93fc-1337b4b52b3b","Type":"ContainerStarted","Data":"3fa35865483c58ea8d152412f74949c131701c54a5a3a2fabf84b5abbe66eeb0"}
Aug 04 17:59:56.582367 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.582313    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd653a884275f59e7e79ba36a5c01dd0" containerID="0f10f4ae70a9bfd1b2e1ab54c7aa9e2fbf435e31e1ed54fae0a69e0d4919fd15" exitCode=0
Aug 04 17:59:56.582572 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.582417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"bd653a884275f59e7e79ba36a5c01dd0","Type":"ContainerDied","Data":"0f10f4ae70a9bfd1b2e1ab54c7aa9e2fbf435e31e1ed54fae0a69e0d4919fd15"}
Aug 04 17:59:56.601520 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.601488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9vhwg" event={"ID":"71e1a397-17f5-4aba-98d7-7a8a0a7be2de","Type":"ContainerStarted","Data":"34d2f5a45ff6b4a8fd01809bb7c998725aeb6d2c1a90a44f592b7f48b9f66ed7"}
Aug 04 17:59:56.604583 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.604555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rl4zg" event={"ID":"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770","Type":"ContainerStarted","Data":"134b9802f6e6b7d415eff1767cd8435374ef37b318d2553f2d939fb41381b33e"}
Aug 04 17:59:56.616252 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.616175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mm2xh" event={"ID":"caccac04-758f-4f30-91d8-d2c828ab88d7","Type":"ContainerStarted","Data":"989998da73aed507021dc357f858f760d2c0e43bd896ea60673d783620309623"}
Aug 04 17:59:56.621593 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.621563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-srl6l" event={"ID":"f067a520-b4c3-47f8-b5af-9eb7ea701266","Type":"ContainerStarted","Data":"9c2211cc89a1753b01071e6faaa9919a45ada423228e235e1f91dd92b2e11f60"}
Aug 04 17:59:56.630725 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.630659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh9j2" event={"ID":"d7e66ed3-d135-4ce6-af93-b463a55a2b6f","Type":"ContainerStarted","Data":"03b06869e782e11274451a710ae26880ca1d0f4577b6fe30dca095656fed7ea5"}
Aug 04 17:59:56.650973 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.650939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" event={"ID":"ced7042c6bb32bfc2b5d6d18b0919b8f","Type":"ContainerStarted","Data":"23ff517dfee87394f1e7257205b6520f29824e44260f598d05c71cd10beb283b"}
Aug 04 17:59:56.656350 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.656237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" event={"ID":"a4ba9272-9ed2-418b-91ce-7f116ee67171","Type":"ContainerStarted","Data":"547fa1f28402e09689ae2247cdaa889aa68e05204a2227156e937927f37bd147"}
Aug 04 17:59:56.669106 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.669045    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" podStartSLOduration=2.669024476 podStartE2EDuration="2.669024476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:56.668227636 +0000 UTC m=+4.602297539" watchObservedRunningTime="2026-08-04 17:59:56.669024476 +0000 UTC m=+4.603094380"
Aug 04 17:59:56.679126 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.679084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp" event={"ID":"1379cd7a-a7b5-44da-9387-490611d058ef","Type":"ContainerStarted","Data":"554fb9ba052d6ae19b0aa84032f617c3a160ee7c5b7ff6205457dce9efd9aa72"}
Aug 04 17:59:56.708893 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:56.708852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sz4dp" event={"ID":"24deb436-b82d-434c-b34b-f85e19cc147f","Type":"ContainerStarted","Data":"37ebef7273ee88e90b97ead1a5da5bd553f9df816e058ec676d9ae0be5d8a616"}
Aug 04 17:59:57.540928 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:57.540892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:57.541449 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:57.541047    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 17:59:57.541551 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:57.541531    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:57.541668 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:57.541646    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 17:59:57.541782 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:57.541739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 17:59:57.541840 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:57.541813    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 17:59:57.722850 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:57.722744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"bd653a884275f59e7e79ba36a5c01dd0","Type":"ContainerStarted","Data":"6099a1cc1296999dd80cadb480cc688ef63ede3a6d1d9a0f89dbc9ab6826a9b1"}
Aug 04 17:59:58.065269 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:58.064374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:58.065269 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:58.064457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:58.065269 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:58.064606    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:58.065269 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:58.064694    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret podName:082c2f9a-47d7-4425-90dc-55ee974b6c58 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:02.064656527 +0000 UTC m=+9.998726406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret") pod "global-pull-secret-syncer-glgg2" (UID: "082c2f9a-47d7-4425-90dc-55ee974b6c58") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:58.065269 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:58.065160    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:58.065269 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:58.065219    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs podName:a8e5dc99-1596-4919-84a4-f8d3bc43ca2d nodeName:}" failed. No retries permitted until 2026-08-04 18:00:02.065200772 +0000 UTC m=+9.999270652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs") pod "network-metrics-daemon-x6dkv" (UID: "a8e5dc99-1596-4919-84a4-f8d3bc43ca2d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:58.166651 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:58.165796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 17:59:58.166651 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:58.166041    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:58.166651 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:58.166062    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:58.166651 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:58.166091    2581 projected.go:196] Error preparing data for projected volume kube-api-access-tnb79 for pod openshift-network-diagnostics/network-check-target-rbnmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:58.166651 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:58.166177    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79 podName:87ce351b-62f5-44a3-89a7-ed4a18221c8a nodeName:}" failed. No retries permitted until 2026-08-04 18:00:02.166135794 +0000 UTC m=+10.100205688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnb79" (UniqueName: "kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79") pod "network-check-target-rbnmz" (UID: "87ce351b-62f5-44a3-89a7-ed4a18221c8a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:59.540943 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:59.540907    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 17:59:59.541437 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:59.541048    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 17:59:59.541437 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:59.541413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 17:59:59.541547 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:59.541513    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 17:59:59.541585 ip-10-0-139-201 kubenswrapper[2581]: I0804 17:59:59.541578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 17:59:59.541712 ip-10-0-139-201 kubenswrapper[2581]: E0804 17:59:59.541691    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:01.540506 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:01.540470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:01.540974 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:01.540636    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:01.540974 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:01.540689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:01.540974 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:01.540769    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:01.540974 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:01.540809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:01.540974 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:01.540859    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:02.098124 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:02.097819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:02.098124 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:02.097925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:02.098124 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:02.097942    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:02.098124 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:02.098004    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret podName:082c2f9a-47d7-4425-90dc-55ee974b6c58 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:10.097988697 +0000 UTC m=+18.032058575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret") pod "global-pull-secret-syncer-glgg2" (UID: "082c2f9a-47d7-4425-90dc-55ee974b6c58") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:02.098124 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:02.098041    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:02.098124 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:02.098095    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs podName:a8e5dc99-1596-4919-84a4-f8d3bc43ca2d nodeName:}" failed. No retries permitted until 2026-08-04 18:00:10.098080192 +0000 UTC m=+18.032150071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs") pod "network-metrics-daemon-x6dkv" (UID: "a8e5dc99-1596-4919-84a4-f8d3bc43ca2d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:02.198776 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:02.198742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:02.199359 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:02.198933    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:02.199359 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:02.198956    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:02.199359 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:02.198969    2581 projected.go:196] Error preparing data for projected volume kube-api-access-tnb79 for pod openshift-network-diagnostics/network-check-target-rbnmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:02.199359 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:02.199030    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79 podName:87ce351b-62f5-44a3-89a7-ed4a18221c8a nodeName:}" failed. No retries permitted until 2026-08-04 18:00:10.19901213 +0000 UTC m=+18.133082022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnb79" (UniqueName: "kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79") pod "network-check-target-rbnmz" (UID: "87ce351b-62f5-44a3-89a7-ed4a18221c8a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:03.540664 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:03.540627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:03.541114 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:03.540763    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:03.541114 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:03.540862    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:03.541114 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:03.540974    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:03.541114 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:03.541038    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:03.541337 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:03.541119    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:05.540276 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:05.540232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:05.540800 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:05.540342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:05.540800 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:05.540355    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:05.540800 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:05.540347    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:05.540800 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:05.540463    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:05.540800 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:05.540554    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:07.540341 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:07.540299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:07.540813 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:07.540316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:07.540813 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:07.540425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:07.540813 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:07.540424    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:07.540813 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:07.540496    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:07.540813 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:07.540627    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:09.540415 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:09.540380    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:09.540855 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:09.540383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:09.540855 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:09.540507    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:09.540855 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:09.540565    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:09.540855 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:09.540383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:09.540855 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:09.540652    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:10.162850 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:10.162814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:10.162850 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:10.162856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:10.163054 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:10.162946    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:10.163054 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:10.162954    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:10.163054 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:10.163015    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret podName:082c2f9a-47d7-4425-90dc-55ee974b6c58 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.162984363 +0000 UTC m=+34.097054241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret") pod "global-pull-secret-syncer-glgg2" (UID: "082c2f9a-47d7-4425-90dc-55ee974b6c58") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:10.163054 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:10.163028    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs podName:a8e5dc99-1596-4919-84a4-f8d3bc43ca2d nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.163021945 +0000 UTC m=+34.097091822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs") pod "network-metrics-daemon-x6dkv" (UID: "a8e5dc99-1596-4919-84a4-f8d3bc43ca2d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:10.263820 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:10.263791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:10.263968 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:10.263945    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:10.263968 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:10.263967    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:10.264037 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:10.263977    2581 projected.go:196] Error preparing data for projected volume kube-api-access-tnb79 for pod openshift-network-diagnostics/network-check-target-rbnmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:10.264037 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:10.264025    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79 podName:87ce351b-62f5-44a3-89a7-ed4a18221c8a nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.264012494 +0000 UTC m=+34.198082371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnb79" (UniqueName: "kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79") pod "network-check-target-rbnmz" (UID: "87ce351b-62f5-44a3-89a7-ed4a18221c8a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:11.539995 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:11.539961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:11.539995 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:11.539990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:11.540444 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:11.539962    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:11.540444 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:11.540117    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:11.540444 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:11.540188    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:11.540444 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:11.540264    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:13.540880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:13.540838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:13.541299 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:13.540838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:13.541299 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:13.540958    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:13.541299 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:13.541041    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:13.541299 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:13.540838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:13.541299 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:13.541138    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:14.760374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.760057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-srl6l" event={"ID":"f067a520-b4c3-47f8-b5af-9eb7ea701266","Type":"ContainerStarted","Data":"8bb1285928149b53ad648f0e034042b0282e1293a40226b301ddf1cd78f23e0b"}
Aug 04 18:00:14.761569 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.761548    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7e66ed3-d135-4ce6-af93-b463a55a2b6f" containerID="7ce7be48f537cdebe59e18867acc2038db680e854e94adc9fb3de8860659a669" exitCode=0
Aug 04 18:00:14.761647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.761627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh9j2" event={"ID":"d7e66ed3-d135-4ce6-af93-b463a55a2b6f","Type":"ContainerDied","Data":"7ce7be48f537cdebe59e18867acc2038db680e854e94adc9fb3de8860659a669"}
Aug 04 18:00:14.763997 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.763977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" event={"ID":"a4ba9272-9ed2-418b-91ce-7f116ee67171","Type":"ContainerStarted","Data":"42763bf2a239a9f7a0ace65a8961cfa29922ab85b0d4257c233cf0d867887661"}
Aug 04 18:00:14.764088 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.764051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" event={"ID":"a4ba9272-9ed2-418b-91ce-7f116ee67171","Type":"ContainerStarted","Data":"30b64f8989f1352565e62abd79aca21546a6a780a641c0222b8f1266c4952232"}
Aug 04 18:00:14.764147 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.764098    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" event={"ID":"a4ba9272-9ed2-418b-91ce-7f116ee67171","Type":"ContainerStarted","Data":"083c76a5459597400220e316eedf2a1b52b4903e4c54d394b44c1c96fdc84197"}
Aug 04 18:00:14.764199 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.764145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" event={"ID":"a4ba9272-9ed2-418b-91ce-7f116ee67171","Type":"ContainerStarted","Data":"9c54419ad4717bf033bc6bc0df46846dfa875211fc570670da969490b86e27f5"}
Aug 04 18:00:14.765431 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.765409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp" event={"ID":"1379cd7a-a7b5-44da-9387-490611d058ef","Type":"ContainerStarted","Data":"9bd99a48120406ba546a4fb8ea28147b0181fb9fab84e5800f7f21974460800a"}
Aug 04 18:00:14.766619 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.766589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sz4dp" event={"ID":"24deb436-b82d-434c-b34b-f85e19cc147f","Type":"ContainerStarted","Data":"8ee0a0fa0ce9e4a5d8fdb38209e9df6633daf5843932b420aaa0f7d54d208940"}
Aug 04 18:00:14.767910 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.767892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rkkz" event={"ID":"1f50134d-2df8-49c9-93fc-1337b4b52b3b","Type":"ContainerStarted","Data":"2cb5b453728f6088e420420ca56a1aba5d43dace6af84a6995c8c56f670ea047"}
Aug 04 18:00:14.769188 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.769169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rl4zg" event={"ID":"1e8bdc52-e2e6-4679-8ff8-de9ba96c3770","Type":"ContainerStarted","Data":"e3a86124b9860133e8a9bc141ec53ab18cb2d2e63cfefd1017d50c93618c2553"}
Aug 04 18:00:14.770358 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.770339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mm2xh" event={"ID":"caccac04-758f-4f30-91d8-d2c828ab88d7","Type":"ContainerStarted","Data":"00c74ae412bb63b9abe93f0ba353f213f2bbc06158955fca63abf7cabd786256"}
Aug 04 18:00:14.778745 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.778707    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-srl6l" podStartSLOduration=3.13442636 podStartE2EDuration="21.778695864s" totalImagesPullingTime="18.644269504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:14.778490521 +0000 UTC m=+22.712560420" watchObservedRunningTime="2026-08-04 18:00:14.778695864 +0000 UTC m=+22.712765764"
Aug 04 18:00:14.779178 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.779156    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" podStartSLOduration=20.779151297 podStartE2EDuration="20.779151297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:57.740954673 +0000 UTC m=+5.675024577" watchObservedRunningTime="2026-08-04 18:00:14.779151297 +0000 UTC m=+22.713221196"
Aug 04 18:00:14.816044 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.816000    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mm2xh" podStartSLOduration=3.238845972 podStartE2EDuration="21.815986159s" totalImagesPullingTime="18.577140187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:14.815302074 +0000 UTC m=+22.749371973" watchObservedRunningTime="2026-08-04 18:00:14.815986159 +0000 UTC m=+22.750056057"
Aug 04 18:00:14.854960 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.854920    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9rkkz" podStartSLOduration=11.163709953 podStartE2EDuration="21.85490561s" totalImagesPullingTime="10.691195657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:14.82971191 +0000 UTC m=+22.763781809" watchObservedRunningTime="2026-08-04 18:00:14.85490561 +0000 UTC m=+22.788975562"
Aug 04 18:00:14.855075 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:14.855038    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sz4dp" podStartSLOduration=3.250160241 podStartE2EDuration="21.855028023s" totalImagesPullingTime="18.604867782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:14.854479029 +0000 UTC m=+22.788548928" watchObservedRunningTime="2026-08-04 18:00:14.855028023 +0000 UTC m=+22.789097924"
Aug 04 18:00:15.540982 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:15.540950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:15.541181 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:15.540989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:15.541181 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:15.541047    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:15.541181 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:15.541058    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:15.541181 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:15.541116    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:15.541181 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:15.541166    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:15.775405 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:15.775320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" event={"ID":"a4ba9272-9ed2-418b-91ce-7f116ee67171","Type":"ContainerStarted","Data":"3f359a250deefc9ce5dabf5d9bfbc092af5bfa260e8a6ce6f5d533d6f641c309"}
Aug 04 18:00:15.775914 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:15.775406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" event={"ID":"a4ba9272-9ed2-418b-91ce-7f116ee67171","Type":"ContainerStarted","Data":"c54277917b74e939c8a8b0a76e255e91d644a47f15d4e85c6abefcc61c66e905"}
Aug 04 18:00:15.776736 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:15.776704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9vhwg" event={"ID":"71e1a397-17f5-4aba-98d7-7a8a0a7be2de","Type":"ContainerStarted","Data":"67eca0d35ff1ac65148e808772a0fa921b2640588b6ff9b4ae619786bfb8e002"}
Aug 04 18:00:15.792963 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:15.792917    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rl4zg" podStartSLOduration=4.215644344 podStartE2EDuration="22.792902615s" totalImagesPullingTime="18.577258271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:14.886174458 +0000 UTC m=+22.820244358" watchObservedRunningTime="2026-08-04 18:00:15.792902615 +0000 UTC m=+23.726972514"
Aug 04 18:00:15.793093 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:15.793057    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9vhwg" podStartSLOduration=4.213537868 podStartE2EDuration="22.793048962s" totalImagesPullingTime="18.579511094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:15.792811483 +0000 UTC m=+23.726881381" watchObservedRunningTime="2026-08-04 18:00:15.793048962 +0000 UTC m=+23.727118863"
Aug 04 18:00:17.330322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:17.330296    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 18:00:17.488421 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:17.488275    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T18:00:17.330315955Z","UUID":"a367bb35-6f96-4a71-b632-790b3c2409c0","Handler":null,"Name":"","Endpoint":""}
Aug 04 18:00:17.489994 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:17.489971    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 18:00:17.490125 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:17.490008    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 18:00:17.540452 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:17.540422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:17.540634 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:17.540514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:17.540634 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:17.540522    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:17.540634 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:17.540540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:17.540634 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:17.540598    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:17.540858 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:17.540710    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:17.784698 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:17.784612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" event={"ID":"a4ba9272-9ed2-418b-91ce-7f116ee67171","Type":"ContainerStarted","Data":"fb8ab908e70a4e2a46856d8a1cf7455082e5a1443233b81322db77109fcb1c10"}
Aug 04 18:00:17.786661 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:17.786634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp" event={"ID":"1379cd7a-a7b5-44da-9387-490611d058ef","Type":"ContainerStarted","Data":"7a420c66e2beb3c0d16970c99db7f38eddad4ef828da5e031737b4b60d08478d"}
Aug 04 18:00:19.540638 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:19.540414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:19.541106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:19.540454    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:19.541106 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:19.540740    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:19.541106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:19.540464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:19.541106 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:19.540807    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:19.541106 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:19.540866    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:19.594961 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:19.594932    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 18:00:19.595671 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:19.595647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 18:00:19.791969 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:19.791880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 18:00:19.792399 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:19.792380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rkkz"
Aug 04 18:00:20.796568 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:20.796308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" event={"ID":"a4ba9272-9ed2-418b-91ce-7f116ee67171","Type":"ContainerStarted","Data":"a613dd7896d3ff400f5f49e1187cf4ebc970ce482a291aa3b1e2f26536bd26fe"}
Aug 04 18:00:20.796568 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:20.796563    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 18:00:20.797392 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:20.796628    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 18:00:20.798671 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:20.798641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp" event={"ID":"1379cd7a-a7b5-44da-9387-490611d058ef","Type":"ContainerStarted","Data":"a984ae0ed59a99a1858c876e02dc974cb08e9f60987121b337eb47941df626b7"}
Aug 04 18:00:20.800198 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:20.800176    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7e66ed3-d135-4ce6-af93-b463a55a2b6f" containerID="89ffb733544b3afb874b07e14f68b8b28da11760742e2085b6bf3fccbc76244f" exitCode=0
Aug 04 18:00:20.800320 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:20.800252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh9j2" event={"ID":"d7e66ed3-d135-4ce6-af93-b463a55a2b6f","Type":"ContainerDied","Data":"89ffb733544b3afb874b07e14f68b8b28da11760742e2085b6bf3fccbc76244f"}
Aug 04 18:00:20.813095 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:20.813079    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 18:00:20.825628 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:20.825578    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr" podStartSLOduration=9.183756827 podStartE2EDuration="27.825567195s" totalImagesPullingTime="18.641810368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:20.825083474 +0000 UTC m=+28.759153504" watchObservedRunningTime="2026-08-04 18:00:20.825567195 +0000 UTC m=+28.759637135"
Aug 04 18:00:20.870306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:20.870264    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxblp" podStartSLOduration=3.8309782 podStartE2EDuration="27.870250624s" totalImagesPullingTime="24.039272424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:20.869749296 +0000 UTC m=+28.803819194" watchObservedRunningTime="2026-08-04 18:00:20.870250624 +0000 UTC m=+28.804320523"
Aug 04 18:00:21.540923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.540894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:21.541091 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.540996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:21.541091 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:21.541003    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:21.541091 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.541047    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:21.541266 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:21.541139    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:21.541266 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:21.541237    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:21.804679 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.804548    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 18:00:21.824641 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.824537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 18:00:21.863026 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.862987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-glgg2"]
Aug 04 18:00:21.863149 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.863108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:21.863212 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:21.863191    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:21.865167 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.865129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x6dkv"]
Aug 04 18:00:21.865300 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.865271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:21.865435 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:21.865410    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:21.865957 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.865847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rbnmz"]
Aug 04 18:00:21.865957 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.865938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:21.866058 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:21.866020    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:21.941256 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:21.941211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkkz_1f50134d-2df8-49c9-93fc-1337b4b52b3b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:22.035013 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:22.034974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkkz_1f50134d-2df8-49c9-93fc-1337b4b52b3b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:22.807441 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:22.807411    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7e66ed3-d135-4ce6-af93-b463a55a2b6f" containerID="2ee9e65fd0649143d2a1dead54451e5bf5dcbd2369593d850e314ebb451fb89d" exitCode=0
Aug 04 18:00:22.807880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:22.807493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh9j2" event={"ID":"d7e66ed3-d135-4ce6-af93-b463a55a2b6f","Type":"ContainerDied","Data":"2ee9e65fd0649143d2a1dead54451e5bf5dcbd2369593d850e314ebb451fb89d"}
Aug 04 18:00:23.540841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:23.540591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:23.540990 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:23.540648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:23.540990 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:23.540866    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:23.540990 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:23.540962    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:23.540990 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:23.540654    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:23.541181 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:23.541099    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:24.250427 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:24.250343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkkz_1f50134d-2df8-49c9-93fc-1337b4b52b3b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:24.368410 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:24.368374    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkkz_1f50134d-2df8-49c9-93fc-1337b4b52b3b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:24.814134 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:24.814101    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7e66ed3-d135-4ce6-af93-b463a55a2b6f" containerID="1632dd53d484e97c52b1ada8bd5c56f89af629e31be73f3abbfaf40ec018ba11" exitCode=0
Aug 04 18:00:24.814275 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:24.814156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh9j2" event={"ID":"d7e66ed3-d135-4ce6-af93-b463a55a2b6f","Type":"ContainerDied","Data":"1632dd53d484e97c52b1ada8bd5c56f89af629e31be73f3abbfaf40ec018ba11"}
Aug 04 18:00:24.923328 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:24.923292    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkkz_1f50134d-2df8-49c9-93fc-1337b4b52b3b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:25.024876 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:25.024841    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkkz_1f50134d-2df8-49c9-93fc-1337b4b52b3b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:25.195261 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:25.195230    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkkz_1f50134d-2df8-49c9-93fc-1337b4b52b3b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:25.295771 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:25.295733    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkkz_1f50134d-2df8-49c9-93fc-1337b4b52b3b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:25.540296 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:25.540211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:25.540296 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:25.540239    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:25.540296 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:25.540255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:25.540546 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:25.540355    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:25.540546 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:25.540444    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:25.540702 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:25.540537    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:26.187930 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:26.187890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:26.188122 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:26.187980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:26.188122 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:26.188045    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:26.188122 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:26.188081    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:26.188122 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:26.188114    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret podName:082c2f9a-47d7-4425-90dc-55ee974b6c58 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:58.188095158 +0000 UTC m=+66.122165040 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret") pod "global-pull-secret-syncer-glgg2" (UID: "082c2f9a-47d7-4425-90dc-55ee974b6c58") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:26.188280 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:26.188137    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs podName:a8e5dc99-1596-4919-84a4-f8d3bc43ca2d nodeName:}" failed. No retries permitted until 2026-08-04 18:00:58.188125658 +0000 UTC m=+66.122195537 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs") pod "network-metrics-daemon-x6dkv" (UID: "a8e5dc99-1596-4919-84a4-f8d3bc43ca2d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:26.289199 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:26.289141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:26.289379 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:26.289352    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:26.289464 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:26.289389    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:26.289464 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:26.289403    2581 projected.go:196] Error preparing data for projected volume kube-api-access-tnb79 for pod openshift-network-diagnostics/network-check-target-rbnmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:26.289552 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:26.289468    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79 podName:87ce351b-62f5-44a3-89a7-ed4a18221c8a nodeName:}" failed. No retries permitted until 2026-08-04 18:00:58.289448506 +0000 UTC m=+66.223518397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnb79" (UniqueName: "kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79") pod "network-check-target-rbnmz" (UID: "87ce351b-62f5-44a3-89a7-ed4a18221c8a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:27.540525 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.540491    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:27.540525 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.540528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:27.541093 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.540492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:27.541093 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:27.540626    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbnmz" podUID="87ce351b-62f5-44a3-89a7-ed4a18221c8a"
Aug 04 18:00:27.541093 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:27.540703    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glgg2" podUID="082c2f9a-47d7-4425-90dc-55ee974b6c58"
Aug 04 18:00:27.541093 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:27.540774    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x6dkv" podUID="a8e5dc99-1596-4919-84a4-f8d3bc43ca2d"
Aug 04 18:00:27.902741 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.902711    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeReady"
Aug 04 18:00:27.902902 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.902846    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 18:00:27.937136 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.937099    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f666cc957-mklk8"]
Aug 04 18:00:27.939799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.939737    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:27.942763 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.942739    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 04 18:00:27.942874 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.942762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-68qht\""
Aug 04 18:00:27.942874 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.942774    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 04 18:00:27.942874 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.942738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 04 18:00:27.947202 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.946963    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mks8f"]
Aug 04 18:00:27.948919 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.948902    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:27.951194 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.951175    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2nvr2"]
Aug 04 18:00:27.951785 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.951619    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 18:00:27.951785 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.951663    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 18:00:27.951785 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.951678    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snn7x\""
Aug 04 18:00:27.952975 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.952957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2nvr2"
Aug 04 18:00:27.956701 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.956679    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 04 18:00:27.958186 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.958166    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 18:00:27.958530 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.958511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 18:00:27.958530 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.958526    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lcczt\""
Aug 04 18:00:27.958889 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.958871    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 18:00:27.962046 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.961886    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f666cc957-mklk8"]
Aug 04 18:00:27.964485 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.964465    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2nvr2"]
Aug 04 18:00:27.967753 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:27.967682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mks8f"]
Aug 04 18:00:28.102286 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec1403a-a186-47a0-8a5d-c7853dca9d20-tmp-dir\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.102286 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-ca-trust-extracted\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.102525 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/350365eb-919f-41fc-aff6-2903c573215b-cert\") pod \"ingress-canary-2nvr2\" (UID: \"350365eb-919f-41fc-aff6-2903c573215b\") " pod="openshift-ingress-canary/ingress-canary-2nvr2"
Aug 04 18:00:28.102525 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-trusted-ca\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.102525 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.102525 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-bound-sa-token\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.102760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ec1403a-a186-47a0-8a5d-c7853dca9d20-config-volume\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.102760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-image-registry-private-configuration\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.102760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-certificates\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.102760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnphp\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-kube-api-access-qnphp\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.102760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrf97\" (UniqueName: \"kubernetes.io/projected/350365eb-919f-41fc-aff6-2903c573215b-kube-api-access-rrf97\") pod \"ingress-canary-2nvr2\" (UID: \"350365eb-919f-41fc-aff6-2903c573215b\") " pod="openshift-ingress-canary/ingress-canary-2nvr2"
Aug 04 18:00:28.102760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbllf\" (UniqueName: \"kubernetes.io/projected/3ec1403a-a186-47a0-8a5d-c7853dca9d20-kube-api-access-rbllf\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.103039 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-installation-pull-secrets\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.103039 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.102803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.203880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.203791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnphp\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-kube-api-access-qnphp\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.203880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.203834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrf97\" (UniqueName: \"kubernetes.io/projected/350365eb-919f-41fc-aff6-2903c573215b-kube-api-access-rrf97\") pod \"ingress-canary-2nvr2\" (UID: \"350365eb-919f-41fc-aff6-2903c573215b\") " pod="openshift-ingress-canary/ingress-canary-2nvr2"
Aug 04 18:00:28.203880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.203857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbllf\" (UniqueName: \"kubernetes.io/projected/3ec1403a-a186-47a0-8a5d-c7853dca9d20-kube-api-access-rbllf\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.203880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.203880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-installation-pull-secrets\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.203908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.203929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec1403a-a186-47a0-8a5d-c7853dca9d20-tmp-dir\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.203950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-ca-trust-extracted\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/350365eb-919f-41fc-aff6-2903c573215b-cert\") pod \"ingress-canary-2nvr2\" (UID: \"350365eb-919f-41fc-aff6-2903c573215b\") " pod="openshift-ingress-canary/ingress-canary-2nvr2"
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-trusted-ca\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.204113    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.204180    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls podName:3ec1403a-a186-47a0-8a5d-c7853dca9d20 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:28.704158832 +0000 UTC m=+36.638228709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls") pod "dns-default-mks8f" (UID: "3ec1403a-a186-47a0-8a5d-c7853dca9d20") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.204232    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.204251    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f666cc957-mklk8: secret "image-registry-tls" not found
Aug 04 18:00:28.204288 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-bound-sa-token\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.204873 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.204318    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls podName:d9982f3c-edf0-4beb-a5ed-9f60e24e475d nodeName:}" failed. No retries permitted until 2026-08-04 18:00:28.704297763 +0000 UTC m=+36.638367661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls") pod "image-registry-7f666cc957-mklk8" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d") : secret "image-registry-tls" not found
Aug 04 18:00:28.204873 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ec1403a-a186-47a0-8a5d-c7853dca9d20-config-volume\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.204873 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-image-registry-private-configuration\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.204873 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-certificates\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.204873 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec1403a-a186-47a0-8a5d-c7853dca9d20-tmp-dir\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.205087 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ec1403a-a186-47a0-8a5d-c7853dca9d20-config-volume\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.205087 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.204963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-ca-trust-extracted\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.205474 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.205450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-trusted-ca\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.205937 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.205915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-certificates\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.208581 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.208561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-installation-pull-secrets\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.208700 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.208579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-image-registry-private-configuration\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.208700 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.208627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/350365eb-919f-41fc-aff6-2903c573215b-cert\") pod \"ingress-canary-2nvr2\" (UID: \"350365eb-919f-41fc-aff6-2903c573215b\") " pod="openshift-ingress-canary/ingress-canary-2nvr2"
Aug 04 18:00:28.213521 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.213498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-bound-sa-token\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.214168 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.213949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrf97\" (UniqueName: \"kubernetes.io/projected/350365eb-919f-41fc-aff6-2903c573215b-kube-api-access-rrf97\") pod \"ingress-canary-2nvr2\" (UID: \"350365eb-919f-41fc-aff6-2903c573215b\") " pod="openshift-ingress-canary/ingress-canary-2nvr2"
Aug 04 18:00:28.214279 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.214240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnphp\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-kube-api-access-qnphp\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.214342 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.214275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbllf\" (UniqueName: \"kubernetes.io/projected/3ec1403a-a186-47a0-8a5d-c7853dca9d20-kube-api-access-rbllf\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.270930 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.270871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2nvr2"
Aug 04 18:00:28.709393 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.709335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:28.710035 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:28.709454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:28.710035 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.709499    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:28.710035 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.709579    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:28.710035 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.709589    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls podName:3ec1403a-a186-47a0-8a5d-c7853dca9d20 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:29.709568234 +0000 UTC m=+37.643638129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls") pod "dns-default-mks8f" (UID: "3ec1403a-a186-47a0-8a5d-c7853dca9d20") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:28.710035 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.709592    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f666cc957-mklk8: secret "image-registry-tls" not found
Aug 04 18:00:28.710035 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:28.709669    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls podName:d9982f3c-edf0-4beb-a5ed-9f60e24e475d nodeName:}" failed. No retries permitted until 2026-08-04 18:00:29.709653022 +0000 UTC m=+37.643722913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls") pod "image-registry-7f666cc957-mklk8" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d") : secret "image-registry-tls" not found
Aug 04 18:00:29.125423 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.125246    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2nvr2"]
Aug 04 18:00:29.128924 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:29.128892    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350365eb_919f_41fc_aff6_2903c573215b.slice/crio-332af9eb1bf9bf45ab5319dd3ac0611b2d774c2dd157cf43d7dea1aa95c8d14e WatchSource:0}: Error finding container 332af9eb1bf9bf45ab5319dd3ac0611b2d774c2dd157cf43d7dea1aa95c8d14e: Status 404 returned error can't find the container with id 332af9eb1bf9bf45ab5319dd3ac0611b2d774c2dd157cf43d7dea1aa95c8d14e
Aug 04 18:00:29.540707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.540619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:29.540707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.540676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:29.540707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.540686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:29.543712 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.543696    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxtnm\""
Aug 04 18:00:29.545420 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.545402    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:00:29.545420 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.545412    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:00:29.545561 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.545445    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:00:29.545561 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.545459    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:00:29.545561 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.545475    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-88fc8\""
Aug 04 18:00:29.717900 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.717867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:29.718339 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.717934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:29.718339 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:29.718036    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:29.718339 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:29.718091    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls podName:3ec1403a-a186-47a0-8a5d-c7853dca9d20 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.718078104 +0000 UTC m=+39.652147980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls") pod "dns-default-mks8f" (UID: "3ec1403a-a186-47a0-8a5d-c7853dca9d20") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:29.718339 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:29.718039    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:29.718339 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:29.718125    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f666cc957-mklk8: secret "image-registry-tls" not found
Aug 04 18:00:29.718339 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:29.718161    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls podName:d9982f3c-edf0-4beb-a5ed-9f60e24e475d nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.71815066 +0000 UTC m=+39.652220544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls") pod "image-registry-7f666cc957-mklk8" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d") : secret "image-registry-tls" not found
Aug 04 18:00:29.828665 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.828578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2nvr2" event={"ID":"350365eb-919f-41fc-aff6-2903c573215b","Type":"ContainerStarted","Data":"332af9eb1bf9bf45ab5319dd3ac0611b2d774c2dd157cf43d7dea1aa95c8d14e"}
Aug 04 18:00:29.831647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.831622    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7e66ed3-d135-4ce6-af93-b463a55a2b6f" containerID="ce4e34afe4091836c20af4f583734fbc576f089e542f06e421f89dbc05d36603" exitCode=0
Aug 04 18:00:29.831782 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:29.831672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh9j2" event={"ID":"d7e66ed3-d135-4ce6-af93-b463a55a2b6f","Type":"ContainerDied","Data":"ce4e34afe4091836c20af4f583734fbc576f089e542f06e421f89dbc05d36603"}
Aug 04 18:00:30.837674 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:30.837632    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7e66ed3-d135-4ce6-af93-b463a55a2b6f" containerID="1be8497906a97145d5fe6463ad408d068cfb33299f9e1526c6e294f012d8eb70" exitCode=0
Aug 04 18:00:30.838165 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:30.837694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh9j2" event={"ID":"d7e66ed3-d135-4ce6-af93-b463a55a2b6f","Type":"ContainerDied","Data":"1be8497906a97145d5fe6463ad408d068cfb33299f9e1526c6e294f012d8eb70"}
Aug 04 18:00:31.732697 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:31.732498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:31.732782 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:31.732669    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:31.732841 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:31.732824    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls podName:3ec1403a-a186-47a0-8a5d-c7853dca9d20 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:35.732805272 +0000 UTC m=+43.666875168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls") pod "dns-default-mks8f" (UID: "3ec1403a-a186-47a0-8a5d-c7853dca9d20") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:31.732841 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:31.732831    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:31.732841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:31.732763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:31.733024 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:31.732843    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f666cc957-mklk8: secret "image-registry-tls" not found
Aug 04 18:00:31.733024 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:31.732956    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls podName:d9982f3c-edf0-4beb-a5ed-9f60e24e475d nodeName:}" failed. No retries permitted until 2026-08-04 18:00:35.732943499 +0000 UTC m=+43.667013391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls") pod "image-registry-7f666cc957-mklk8" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d") : secret "image-registry-tls" not found
Aug 04 18:00:31.843672 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:31.843639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh9j2" event={"ID":"d7e66ed3-d135-4ce6-af93-b463a55a2b6f","Type":"ContainerStarted","Data":"c1c3e0693b5ee99bdcd81378fd8a3000a977ccc29742bce6670daa6512f9216c"}
Aug 04 18:00:31.845149 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:31.845129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2nvr2" event={"ID":"350365eb-919f-41fc-aff6-2903c573215b","Type":"ContainerStarted","Data":"733ff7ee0f697736dd8cd0de36db750ecc3b29cf437ebc235a53dd853be1e937"}
Aug 04 18:00:31.870578 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:31.870533    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lh9j2" podStartSLOduration=8.749194212999999 podStartE2EDuration="38.870517397s" totalImagesPullingTime="30.121323184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:31.869242807 +0000 UTC m=+39.803312707" watchObservedRunningTime="2026-08-04 18:00:31.870517397 +0000 UTC m=+39.804587289"
Aug 04 18:00:31.886593 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:31.886548    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2nvr2" podStartSLOduration=2.329996173 podStartE2EDuration="4.886532011s" totalImagesPullingTime="2.556535838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:31.886257706 +0000 UTC m=+39.820327607" watchObservedRunningTime="2026-08-04 18:00:31.886532011 +0000 UTC m=+39.820601911"
Aug 04 18:00:33.071816 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.071786    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nqb98"]
Aug 04 18:00:33.105229 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.105197    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nqb98"]
Aug 04 18:00:33.105371 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.105300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.109559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.109538    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 18:00:33.109559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.109550    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 18:00:33.109559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.109543    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 18:00:33.109802 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.109558    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 18:00:33.111844 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.111829    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sm4mb\""
Aug 04 18:00:33.142012 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.141990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.142117 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.142024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.142117 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.142057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fb0db2a-5076-4279-b371-c8af54f232b6-crio-socket\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.142184 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.142120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fb0db2a-5076-4279-b371-c8af54f232b6-data-volume\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.142184 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.142136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmj5q\" (UniqueName: \"kubernetes.io/projected/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-api-access-fmj5q\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.243044 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.243013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fb0db2a-5076-4279-b371-c8af54f232b6-crio-socket\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.243044 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.243060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fb0db2a-5076-4279-b371-c8af54f232b6-data-volume\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.243287 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.243095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmj5q\" (UniqueName: \"kubernetes.io/projected/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-api-access-fmj5q\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.243287 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.243171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.243287 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.243209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.243287 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.243223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fb0db2a-5076-4279-b371-c8af54f232b6-crio-socket\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.243287 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:33.243274    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:33.243287 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:33.243285    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:33.243580 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:33.243342    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls podName:6fb0db2a-5076-4279-b371-c8af54f232b6 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:33.743322256 +0000 UTC m=+41.677392133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqb98" (UID: "6fb0db2a-5076-4279-b371-c8af54f232b6") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:33.243580 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:33.243363    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm podName:6fb0db2a-5076-4279-b371-c8af54f232b6 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:33.743353226 +0000 UTC m=+41.677423108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nqb98" (UID: "6fb0db2a-5076-4279-b371-c8af54f232b6") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:33.243580 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.243454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fb0db2a-5076-4279-b371-c8af54f232b6-data-volume\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.261348 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.261324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmj5q\" (UniqueName: \"kubernetes.io/projected/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-api-access-fmj5q\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.746534 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.746495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.746749 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:33.746543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:33.746749 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:33.746645    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:33.746749 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:33.746649    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:33.746749 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:33.746708    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm podName:6fb0db2a-5076-4279-b371-c8af54f232b6 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.746691007 +0000 UTC m=+42.680760885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nqb98" (UID: "6fb0db2a-5076-4279-b371-c8af54f232b6") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:33.746749 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:33.746725    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls podName:6fb0db2a-5076-4279-b371-c8af54f232b6 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.746716686 +0000 UTC m=+42.680786566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqb98" (UID: "6fb0db2a-5076-4279-b371-c8af54f232b6") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:34.286753 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.286723    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6"]
Aug 04 18:00:34.302540 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.302505    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6"]
Aug 04 18:00:34.302710 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.302620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6"
Aug 04 18:00:34.305524 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.305503    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:34.306731 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.306714    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4wpxt\""
Aug 04 18:00:34.306926 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.306911    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Aug 04 18:00:34.350738 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.350705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr8gl\" (UniqueName: \"kubernetes.io/projected/8e2d2773-14f9-42c9-8d3b-a4573c0f800d-kube-api-access-dr8gl\") pod \"migrator-68bc745d87-zsqt6\" (UID: \"8e2d2773-14f9-42c9-8d3b-a4573c0f800d\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6"
Aug 04 18:00:34.451507 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.451472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr8gl\" (UniqueName: \"kubernetes.io/projected/8e2d2773-14f9-42c9-8d3b-a4573c0f800d-kube-api-access-dr8gl\") pod \"migrator-68bc745d87-zsqt6\" (UID: \"8e2d2773-14f9-42c9-8d3b-a4573c0f800d\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6"
Aug 04 18:00:34.460877 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.460858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr8gl\" (UniqueName: \"kubernetes.io/projected/8e2d2773-14f9-42c9-8d3b-a4573c0f800d-kube-api-access-dr8gl\") pod \"migrator-68bc745d87-zsqt6\" (UID: \"8e2d2773-14f9-42c9-8d3b-a4573c0f800d\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6"
Aug 04 18:00:34.602555 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.602479    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rl4zg_1e8bdc52-e2e6-4679-8ff8-de9ba96c3770/dns-node-resolver/0.log"
Aug 04 18:00:34.611615 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.611584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6"
Aug 04 18:00:34.748520 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.748483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6"]
Aug 04 18:00:34.753117 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:34.753091    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2d2773_14f9_42c9_8d3b_a4573c0f800d.slice/crio-9493d6771ec50a2609441ae7ff77340c975f941a18c5b09ed7594ab26eb99c28 WatchSource:0}: Error finding container 9493d6771ec50a2609441ae7ff77340c975f941a18c5b09ed7594ab26eb99c28: Status 404 returned error can't find the container with id 9493d6771ec50a2609441ae7ff77340c975f941a18c5b09ed7594ab26eb99c28
Aug 04 18:00:34.753207 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.753153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:34.753207 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.753193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:34.753298 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:34.753286    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:34.753353 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:34.753344    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls podName:6fb0db2a-5076-4279-b371-c8af54f232b6 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:36.753331317 +0000 UTC m=+44.687401194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqb98" (UID: "6fb0db2a-5076-4279-b371-c8af54f232b6") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:34.753406 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:34.753288    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:34.753439 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:34.753422    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm podName:6fb0db2a-5076-4279-b371-c8af54f232b6 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:36.753409438 +0000 UTC m=+44.687479315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nqb98" (UID: "6fb0db2a-5076-4279-b371-c8af54f232b6") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:34.852767 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:34.852685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6" event={"ID":"8e2d2773-14f9-42c9-8d3b-a4573c0f800d","Type":"ContainerStarted","Data":"9493d6771ec50a2609441ae7ff77340c975f941a18c5b09ed7594ab26eb99c28"}
Aug 04 18:00:35.064330 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.064300    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-gn2qn"]
Aug 04 18:00:35.094868 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.094844    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-gn2qn"]
Aug 04 18:00:35.095013 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.094930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.097990 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.097958    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Aug 04 18:00:35.098100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.098055    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Aug 04 18:00:35.098100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.098090    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bx6lg\""
Aug 04 18:00:35.099266 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.099243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Aug 04 18:00:35.099357 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.099314    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Aug 04 18:00:35.155008 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.154981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c3484be-eef6-451d-8bf1-e38ad3533a0e-signing-cabundle\") pod \"service-ca-98c88fbb5-gn2qn\" (UID: \"9c3484be-eef6-451d-8bf1-e38ad3533a0e\") " pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.155125 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.155052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c3484be-eef6-451d-8bf1-e38ad3533a0e-signing-key\") pod \"service-ca-98c88fbb5-gn2qn\" (UID: \"9c3484be-eef6-451d-8bf1-e38ad3533a0e\") " pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.155125 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.155072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rmn\" (UniqueName: \"kubernetes.io/projected/9c3484be-eef6-451d-8bf1-e38ad3533a0e-kube-api-access-28rmn\") pod \"service-ca-98c88fbb5-gn2qn\" (UID: \"9c3484be-eef6-451d-8bf1-e38ad3533a0e\") " pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.256253 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.256223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c3484be-eef6-451d-8bf1-e38ad3533a0e-signing-cabundle\") pod \"service-ca-98c88fbb5-gn2qn\" (UID: \"9c3484be-eef6-451d-8bf1-e38ad3533a0e\") " pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.256410 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.256332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c3484be-eef6-451d-8bf1-e38ad3533a0e-signing-key\") pod \"service-ca-98c88fbb5-gn2qn\" (UID: \"9c3484be-eef6-451d-8bf1-e38ad3533a0e\") " pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.256410 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.256359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rmn\" (UniqueName: \"kubernetes.io/projected/9c3484be-eef6-451d-8bf1-e38ad3533a0e-kube-api-access-28rmn\") pod \"service-ca-98c88fbb5-gn2qn\" (UID: \"9c3484be-eef6-451d-8bf1-e38ad3533a0e\") " pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.256995 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.256974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c3484be-eef6-451d-8bf1-e38ad3533a0e-signing-cabundle\") pod \"service-ca-98c88fbb5-gn2qn\" (UID: \"9c3484be-eef6-451d-8bf1-e38ad3533a0e\") " pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.258685 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.258668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c3484be-eef6-451d-8bf1-e38ad3533a0e-signing-key\") pod \"service-ca-98c88fbb5-gn2qn\" (UID: \"9c3484be-eef6-451d-8bf1-e38ad3533a0e\") " pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.265680 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.265658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rmn\" (UniqueName: \"kubernetes.io/projected/9c3484be-eef6-451d-8bf1-e38ad3533a0e-kube-api-access-28rmn\") pod \"service-ca-98c88fbb5-gn2qn\" (UID: \"9c3484be-eef6-451d-8bf1-e38ad3533a0e\") " pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.402593 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.402560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mm2xh_caccac04-758f-4f30-91d8-d2c828ab88d7/node-ca/0.log"
Aug 04 18:00:35.403681 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.403657    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn"
Aug 04 18:00:35.556380 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.556339    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-gn2qn"]
Aug 04 18:00:35.559678 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:35.559646    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3484be_eef6_451d_8bf1_e38ad3533a0e.slice/crio-4fb4776492a75cc620edbce2b8356c9819974498429574ccf7fc25f9c50a1eb9 WatchSource:0}: Error finding container 4fb4776492a75cc620edbce2b8356c9819974498429574ccf7fc25f9c50a1eb9: Status 404 returned error can't find the container with id 4fb4776492a75cc620edbce2b8356c9819974498429574ccf7fc25f9c50a1eb9
Aug 04 18:00:35.761023 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.760939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:35.761023 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.761019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:35.761243 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:35.761117    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:35.761243 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:35.761140    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f666cc957-mklk8: secret "image-registry-tls" not found
Aug 04 18:00:35.761243 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:35.761128    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:35.761243 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:35.761211    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls podName:d9982f3c-edf0-4beb-a5ed-9f60e24e475d nodeName:}" failed. No retries permitted until 2026-08-04 18:00:43.761190705 +0000 UTC m=+51.695260599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls") pod "image-registry-7f666cc957-mklk8" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d") : secret "image-registry-tls" not found
Aug 04 18:00:35.761438 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:35.761250    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls podName:3ec1403a-a186-47a0-8a5d-c7853dca9d20 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:43.761234784 +0000 UTC m=+51.695304681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls") pod "dns-default-mks8f" (UID: "3ec1403a-a186-47a0-8a5d-c7853dca9d20") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:35.855697 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:35.855662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn" event={"ID":"9c3484be-eef6-451d-8bf1-e38ad3533a0e","Type":"ContainerStarted","Data":"4fb4776492a75cc620edbce2b8356c9819974498429574ccf7fc25f9c50a1eb9"}
Aug 04 18:00:36.212200 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:36.212168    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2nvr2_350365eb-919f-41fc-aff6-2903c573215b/serve-healthcheck-canary/0.log"
Aug 04 18:00:36.770233 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:36.770162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:36.770233 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:36.770200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:36.770648 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:36.770275    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:36.770648 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:36.770301    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:36.770648 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:36.770326    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm podName:6fb0db2a-5076-4279-b371-c8af54f232b6 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:40.770313407 +0000 UTC m=+48.704383284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nqb98" (UID: "6fb0db2a-5076-4279-b371-c8af54f232b6") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:36.770648 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:36.770346    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls podName:6fb0db2a-5076-4279-b371-c8af54f232b6 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:40.770333668 +0000 UTC m=+48.704403545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqb98" (UID: "6fb0db2a-5076-4279-b371-c8af54f232b6") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:36.860196 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:36.860162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6" event={"ID":"8e2d2773-14f9-42c9-8d3b-a4573c0f800d","Type":"ContainerStarted","Data":"b0f2117b4bd7c860cbbd5cb7c765ac780fc4a8b0a25c526b3dd33c3fd924b18d"}
Aug 04 18:00:36.860355 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:36.860222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6" event={"ID":"8e2d2773-14f9-42c9-8d3b-a4573c0f800d","Type":"ContainerStarted","Data":"9d27219ebe6310ba96de530f8d74fab4dae161061d8c34d54163985a85f278e9"}
Aug 04 18:00:36.886527 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:36.886470    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zsqt6" podStartSLOduration=1.12058338 podStartE2EDuration="2.886450081s" totalImagesPullingTime="1.765866701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:36.885987089 +0000 UTC m=+44.820056992" watchObservedRunningTime="2026-08-04 18:00:36.886450081 +0000 UTC m=+44.820519984"
Aug 04 18:00:37.864468 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:37.864433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn" event={"ID":"9c3484be-eef6-451d-8bf1-e38ad3533a0e","Type":"ContainerStarted","Data":"f83c2724df05d1e99bbae16f76b65793ceeb876098c36e597932582489f4868b"}
Aug 04 18:00:37.882524 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:37.882466    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-gn2qn" podStartSLOduration=0.715689436 podStartE2EDuration="2.882446946s" totalImagesPullingTime="2.16675751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:37.881344979 +0000 UTC m=+45.815414878" watchObservedRunningTime="2026-08-04 18:00:37.882446946 +0000 UTC m=+45.816516846"
Aug 04 18:00:40.806241 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:40.806202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:40.806635 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:40.806267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:40.806635 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:40.806353    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:40.806635 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:00:40.806434    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls podName:6fb0db2a-5076-4279-b371-c8af54f232b6 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:48.806413885 +0000 UTC m=+56.740483777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqb98" (UID: "6fb0db2a-5076-4279-b371-c8af54f232b6") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:40.806871 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:40.806854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fb0db2a-5076-4279-b371-c8af54f232b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:43.834613 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:43.834558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:43.835082 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:43.834652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:43.836923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:43.836901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ec1403a-a186-47a0-8a5d-c7853dca9d20-metrics-tls\") pod \"dns-default-mks8f\" (UID: \"3ec1403a-a186-47a0-8a5d-c7853dca9d20\") " pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:43.836991 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:43.836949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") pod \"image-registry-7f666cc957-mklk8\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") " pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:43.850862 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:43.850837    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:43.862137 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:43.862112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:44.001519 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:44.001496    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f666cc957-mklk8"]
Aug 04 18:00:44.004039 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:44.004010    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9982f3c_edf0_4beb_a5ed_9f60e24e475d.slice/crio-6adccd4daea52ed4765e902b8c341892d9e3a6b4562ce503e2e72c2fbac8e612 WatchSource:0}: Error finding container 6adccd4daea52ed4765e902b8c341892d9e3a6b4562ce503e2e72c2fbac8e612: Status 404 returned error can't find the container with id 6adccd4daea52ed4765e902b8c341892d9e3a6b4562ce503e2e72c2fbac8e612
Aug 04 18:00:44.017424 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:44.017400    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mks8f"]
Aug 04 18:00:44.020252 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:44.020227    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec1403a_a186_47a0_8a5d_c7853dca9d20.slice/crio-6e34d7a54a26218c0a6c2cf28dfaba8deee8a23a0f989a95d2d836203290f739 WatchSource:0}: Error finding container 6e34d7a54a26218c0a6c2cf28dfaba8deee8a23a0f989a95d2d836203290f739: Status 404 returned error can't find the container with id 6e34d7a54a26218c0a6c2cf28dfaba8deee8a23a0f989a95d2d836203290f739
Aug 04 18:00:44.885522 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:44.885480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mks8f" event={"ID":"3ec1403a-a186-47a0-8a5d-c7853dca9d20","Type":"ContainerStarted","Data":"6e34d7a54a26218c0a6c2cf28dfaba8deee8a23a0f989a95d2d836203290f739"}
Aug 04 18:00:44.887225 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:44.887192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f666cc957-mklk8" event={"ID":"d9982f3c-edf0-4beb-a5ed-9f60e24e475d","Type":"ContainerStarted","Data":"86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4"}
Aug 04 18:00:44.887365 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:44.887246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f666cc957-mklk8" event={"ID":"d9982f3c-edf0-4beb-a5ed-9f60e24e475d","Type":"ContainerStarted","Data":"6adccd4daea52ed4765e902b8c341892d9e3a6b4562ce503e2e72c2fbac8e612"}
Aug 04 18:00:44.887365 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:44.887351    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:00:44.910254 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:44.910204    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f666cc957-mklk8" podStartSLOduration=33.910187397 podStartE2EDuration="33.910187397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:44.909005252 +0000 UTC m=+52.843075151" watchObservedRunningTime="2026-08-04 18:00:44.910187397 +0000 UTC m=+52.844257300"
Aug 04 18:00:46.895252 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:46.895219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mks8f" event={"ID":"3ec1403a-a186-47a0-8a5d-c7853dca9d20","Type":"ContainerStarted","Data":"1921ac95e5a57c7ea85565777a7a5453ccad63f90aef8353a3ad11619ca141f3"}
Aug 04 18:00:46.895666 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:46.895271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mks8f" event={"ID":"3ec1403a-a186-47a0-8a5d-c7853dca9d20","Type":"ContainerStarted","Data":"77704c20e6ed68243c277a61ca31a547fe0d637ca0c404d3fc04fe62b20384cb"}
Aug 04 18:00:46.895666 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:46.895336    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:46.914729 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:46.914676    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mks8f" podStartSLOduration=17.82361565 podStartE2EDuration="19.914660639s" totalImagesPullingTime="2.091044989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:46.914064787 +0000 UTC m=+54.848134686" watchObservedRunningTime="2026-08-04 18:00:46.914660639 +0000 UTC m=+54.848730532"
Aug 04 18:00:48.877529 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:48.877477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:48.879719 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:48.879697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fb0db2a-5076-4279-b371-c8af54f232b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqb98\" (UID: \"6fb0db2a-5076-4279-b371-c8af54f232b6\") " pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:49.014304 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:49.014268    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nqb98"
Aug 04 18:00:49.136033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:49.136009    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nqb98"]
Aug 04 18:00:49.138447 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:49.138427    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb0db2a_5076_4279_b371_c8af54f232b6.slice/crio-697c38896cc25ca16ba80536ef154aa5a29bcf523ce1e838e952c5c91193d969 WatchSource:0}: Error finding container 697c38896cc25ca16ba80536ef154aa5a29bcf523ce1e838e952c5c91193d969: Status 404 returned error can't find the container with id 697c38896cc25ca16ba80536ef154aa5a29bcf523ce1e838e952c5c91193d969
Aug 04 18:00:49.909382 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:49.909342    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nqb98" event={"ID":"6fb0db2a-5076-4279-b371-c8af54f232b6","Type":"ContainerStarted","Data":"b3f5e460bc173e3a2e4c7cf5b5902d2c989ac62b86750798369a8fb9d90de7ae"}
Aug 04 18:00:49.909836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:49.909407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nqb98" event={"ID":"6fb0db2a-5076-4279-b371-c8af54f232b6","Type":"ContainerStarted","Data":"697c38896cc25ca16ba80536ef154aa5a29bcf523ce1e838e952c5c91193d969"}
Aug 04 18:00:50.914943 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:50.914905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nqb98" event={"ID":"6fb0db2a-5076-4279-b371-c8af54f232b6","Type":"ContainerStarted","Data":"0607aa68994dfa58f75d4cb84b59d844c10dc8236b76185341b48b6de723fb09"}
Aug 04 18:00:52.922194 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:52.922156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nqb98" event={"ID":"6fb0db2a-5076-4279-b371-c8af54f232b6","Type":"ContainerStarted","Data":"4e92231feb00ae986e12c6a80ece3b79e9dddf4221c163a5b3a36fec149b29a2"}
Aug 04 18:00:52.942359 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:52.942308    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nqb98" podStartSLOduration=17.266779825 podStartE2EDuration="19.942292729s" totalImagesPullingTime="2.675512904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:52.94179968 +0000 UTC m=+60.875869578" watchObservedRunningTime="2026-08-04 18:00:52.942292729 +0000 UTC m=+60.876362628"
Aug 04 18:00:53.820316 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:53.820289    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ncgkr"
Aug 04 18:00:56.903063 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:56.903029    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mks8f"
Aug 04 18:00:57.026596 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.026565    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"]
Aug 04 18:00:57.029820 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.029801    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.034485 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.034454    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Aug 04 18:00:57.034586 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.034509    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 04 18:00:57.034586 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.034455    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 04 18:00:57.034586 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.034457    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 04 18:00:57.038501 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.038482    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"]
Aug 04 18:00:57.040271 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.040251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"
Aug 04 18:00:57.043274 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.043253    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jtczh\""
Aug 04 18:00:57.043839 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.043815    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"]
Aug 04 18:00:57.044014 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.043994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Aug 04 18:00:57.046221 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.046106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.048390 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.048372    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"]
Aug 04 18:00:57.049510 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.049488    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Aug 04 18:00:57.049686 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.049488    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Aug 04 18:00:57.049686 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.049591    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Aug 04 18:00:57.049872 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.049858    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Aug 04 18:00:57.057760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.057739    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f666cc957-mklk8"]
Aug 04 18:00:57.058670 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.058650    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"]
Aug 04 18:00:57.067832 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.067803    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"]
Aug 04 18:00:57.123997 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.123965    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-599d78bb5d-4dgsq"]
Aug 04 18:00:57.125864 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.125849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.139003 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.138978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ff2119f9-d2f0-44b3-a5e7-71990b0985b1-klusterlet-config\") pod \"klusterlet-addon-workmgr-74c4574c57-r5t6d\" (UID: \"ff2119f9-d2f0-44b3-a5e7-71990b0985b1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.139127 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8n8v\" (UniqueName: \"kubernetes.io/projected/ff2119f9-d2f0-44b3-a5e7-71990b0985b1-kube-api-access-t8n8v\") pod \"klusterlet-addon-workmgr-74c4574c57-r5t6d\" (UID: \"ff2119f9-d2f0-44b3-a5e7-71990b0985b1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.139127 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.139127 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-hub\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.139127 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqs9\" (UniqueName: \"kubernetes.io/projected/51ab395f-3330-4c91-a828-d2a72c636844-kube-api-access-wqqs9\") pod \"managed-serviceaccount-addon-agent-8464fdc88-wbxxt\" (UID: \"51ab395f-3330-4c91-a828-d2a72c636844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"
Aug 04 18:00:57.139277 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcz7v\" (UniqueName: \"kubernetes.io/projected/4c173588-249a-419c-850f-d79e5f1ae463-kube-api-access-vcz7v\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.139277 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51ab395f-3330-4c91-a828-d2a72c636844-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8464fdc88-wbxxt\" (UID: \"51ab395f-3330-4c91-a828-d2a72c636844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"
Aug 04 18:00:57.139277 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c173588-249a-419c-850f-d79e5f1ae463-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.139277 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-ca\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.139407 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.139407 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.139301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff2119f9-d2f0-44b3-a5e7-71990b0985b1-tmp\") pod \"klusterlet-addon-workmgr-74c4574c57-r5t6d\" (UID: \"ff2119f9-d2f0-44b3-a5e7-71990b0985b1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.148129 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.148078    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-599d78bb5d-4dgsq"]
Aug 04 18:00:57.240109 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50848535-20e5-443f-9da0-1132f2280608-installation-pull-secrets\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.240109 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z5f5\" (UniqueName: \"kubernetes.io/projected/50848535-20e5-443f-9da0-1132f2280608-kube-api-access-7z5f5\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.240109 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ff2119f9-d2f0-44b3-a5e7-71990b0985b1-klusterlet-config\") pod \"klusterlet-addon-workmgr-74c4574c57-r5t6d\" (UID: \"ff2119f9-d2f0-44b3-a5e7-71990b0985b1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.240381 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50848535-20e5-443f-9da0-1132f2280608-registry-tls\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.240381 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8n8v\" (UniqueName: \"kubernetes.io/projected/ff2119f9-d2f0-44b3-a5e7-71990b0985b1-kube-api-access-t8n8v\") pod \"klusterlet-addon-workmgr-74c4574c57-r5t6d\" (UID: \"ff2119f9-d2f0-44b3-a5e7-71990b0985b1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.240381 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.240381 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-hub\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.240381 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqs9\" (UniqueName: \"kubernetes.io/projected/51ab395f-3330-4c91-a828-d2a72c636844-kube-api-access-wqqs9\") pod \"managed-serviceaccount-addon-agent-8464fdc88-wbxxt\" (UID: \"51ab395f-3330-4c91-a828-d2a72c636844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"
Aug 04 18:00:57.240638 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50848535-20e5-443f-9da0-1132f2280608-bound-sa-token\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.240638 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcz7v\" (UniqueName: \"kubernetes.io/projected/4c173588-249a-419c-850f-d79e5f1ae463-kube-api-access-vcz7v\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.240638 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50848535-20e5-443f-9da0-1132f2280608-image-registry-private-configuration\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.240638 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51ab395f-3330-4c91-a828-d2a72c636844-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8464fdc88-wbxxt\" (UID: \"51ab395f-3330-4c91-a828-d2a72c636844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"
Aug 04 18:00:57.240638 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c173588-249a-419c-850f-d79e5f1ae463-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.240638 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50848535-20e5-443f-9da0-1132f2280608-trusted-ca\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.240638 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.240581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50848535-20e5-443f-9da0-1132f2280608-ca-trust-extracted\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.241179 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.241153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50848535-20e5-443f-9da0-1132f2280608-registry-certificates\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.241277 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.241215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-ca\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.241277 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.241266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.241393 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.241305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff2119f9-d2f0-44b3-a5e7-71990b0985b1-tmp\") pod \"klusterlet-addon-workmgr-74c4574c57-r5t6d\" (UID: \"ff2119f9-d2f0-44b3-a5e7-71990b0985b1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.241785 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.241758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff2119f9-d2f0-44b3-a5e7-71990b0985b1-tmp\") pod \"klusterlet-addon-workmgr-74c4574c57-r5t6d\" (UID: \"ff2119f9-d2f0-44b3-a5e7-71990b0985b1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.241882 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.241805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c173588-249a-419c-850f-d79e5f1ae463-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.243706 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.243226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ff2119f9-d2f0-44b3-a5e7-71990b0985b1-klusterlet-config\") pod \"klusterlet-addon-workmgr-74c4574c57-r5t6d\" (UID: \"ff2119f9-d2f0-44b3-a5e7-71990b0985b1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.243794 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.243720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.244773 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.244302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-ca\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.247760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.247737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.247919 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.247897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4c173588-249a-419c-850f-d79e5f1ae463-hub\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.248401 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.248378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51ab395f-3330-4c91-a828-d2a72c636844-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8464fdc88-wbxxt\" (UID: \"51ab395f-3330-4c91-a828-d2a72c636844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"
Aug 04 18:00:57.253833 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.253815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcz7v\" (UniqueName: \"kubernetes.io/projected/4c173588-249a-419c-850f-d79e5f1ae463-kube-api-access-vcz7v\") pod \"cluster-proxy-proxy-agent-5bcb8b498d-sr9f4\" (UID: \"4c173588-249a-419c-850f-d79e5f1ae463\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.254004 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.253987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqs9\" (UniqueName: \"kubernetes.io/projected/51ab395f-3330-4c91-a828-d2a72c636844-kube-api-access-wqqs9\") pod \"managed-serviceaccount-addon-agent-8464fdc88-wbxxt\" (UID: \"51ab395f-3330-4c91-a828-d2a72c636844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"
Aug 04 18:00:57.254426 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.254406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8n8v\" (UniqueName: \"kubernetes.io/projected/ff2119f9-d2f0-44b3-a5e7-71990b0985b1-kube-api-access-t8n8v\") pod \"klusterlet-addon-workmgr-74c4574c57-r5t6d\" (UID: \"ff2119f9-d2f0-44b3-a5e7-71990b0985b1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.339069 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.339035    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:00:57.342652 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.342633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50848535-20e5-443f-9da0-1132f2280608-registry-certificates\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.342748 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.342697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50848535-20e5-443f-9da0-1132f2280608-installation-pull-secrets\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.342748 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.342727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z5f5\" (UniqueName: \"kubernetes.io/projected/50848535-20e5-443f-9da0-1132f2280608-kube-api-access-7z5f5\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.342854 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.342767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50848535-20e5-443f-9da0-1132f2280608-registry-tls\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.342854 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.342815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50848535-20e5-443f-9da0-1132f2280608-bound-sa-token\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.342952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.342855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50848535-20e5-443f-9da0-1132f2280608-image-registry-private-configuration\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.342952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.342885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50848535-20e5-443f-9da0-1132f2280608-trusted-ca\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.342952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.342913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50848535-20e5-443f-9da0-1132f2280608-ca-trust-extracted\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.343324 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.343296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50848535-20e5-443f-9da0-1132f2280608-ca-trust-extracted\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.343553 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.343529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50848535-20e5-443f-9da0-1132f2280608-registry-certificates\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.343993 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.343975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50848535-20e5-443f-9da0-1132f2280608-trusted-ca\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.345222 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.345201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50848535-20e5-443f-9da0-1132f2280608-image-registry-private-configuration\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.345579 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.345563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50848535-20e5-443f-9da0-1132f2280608-registry-tls\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.345894 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.345876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50848535-20e5-443f-9da0-1132f2280608-installation-pull-secrets\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.354538 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.354517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50848535-20e5-443f-9da0-1132f2280608-bound-sa-token\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.354928 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.354906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z5f5\" (UniqueName: \"kubernetes.io/projected/50848535-20e5-443f-9da0-1132f2280608-kube-api-access-7z5f5\") pod \"image-registry-599d78bb5d-4dgsq\" (UID: \"50848535-20e5-443f-9da0-1132f2280608\") " pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.359693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.359675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"
Aug 04 18:00:57.365222 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.365175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"
Aug 04 18:00:57.453858 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.453529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.478663 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.478525    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"]
Aug 04 18:00:57.483450 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:57.483420    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2119f9_d2f0_44b3_a5e7_71990b0985b1.slice/crio-f2fe5f8ed02b1c3fc10d9daf78d9a39a0dbbcfd4ccdf00b05b3d4e33d4bb5a1f WatchSource:0}: Error finding container f2fe5f8ed02b1c3fc10d9daf78d9a39a0dbbcfd4ccdf00b05b3d4e33d4bb5a1f: Status 404 returned error can't find the container with id f2fe5f8ed02b1c3fc10d9daf78d9a39a0dbbcfd4ccdf00b05b3d4e33d4bb5a1f
Aug 04 18:00:57.507414 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.507394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4"]
Aug 04 18:00:57.509754 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:57.509727    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c173588_249a_419c_850f_d79e5f1ae463.slice/crio-61397c8c6f6149571abbad8943c23fa5640320f524eba99a084cc73f651f11b9 WatchSource:0}: Error finding container 61397c8c6f6149571abbad8943c23fa5640320f524eba99a084cc73f651f11b9: Status 404 returned error can't find the container with id 61397c8c6f6149571abbad8943c23fa5640320f524eba99a084cc73f651f11b9
Aug 04 18:00:57.520231 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.520205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt"]
Aug 04 18:00:57.524575 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:57.524548    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51ab395f_3330_4c91_a828_d2a72c636844.slice/crio-eb40c5e5b82108a76265da329d2e52873748bb58f91f2f4a64ad4195d2db0e05 WatchSource:0}: Error finding container eb40c5e5b82108a76265da329d2e52873748bb58f91f2f4a64ad4195d2db0e05: Status 404 returned error can't find the container with id eb40c5e5b82108a76265da329d2e52873748bb58f91f2f4a64ad4195d2db0e05
Aug 04 18:00:57.599341 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.599313    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-599d78bb5d-4dgsq"]
Aug 04 18:00:57.603232 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:57.603199    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50848535_20e5_443f_9da0_1132f2280608.slice/crio-6bd8227b35f266a47ae7a5d12788881679524949e71ab8b633d398caf7257d8d WatchSource:0}: Error finding container 6bd8227b35f266a47ae7a5d12788881679524949e71ab8b633d398caf7257d8d: Status 404 returned error can't find the container with id 6bd8227b35f266a47ae7a5d12788881679524949e71ab8b633d398caf7257d8d
Aug 04 18:00:57.938934 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.938893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq" event={"ID":"50848535-20e5-443f-9da0-1132f2280608","Type":"ContainerStarted","Data":"ce77ffca3ffb1e80e580226d7eeb759c58cec321c0758381cd13957551bc56e5"}
Aug 04 18:00:57.939401 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.938952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq" event={"ID":"50848535-20e5-443f-9da0-1132f2280608","Type":"ContainerStarted","Data":"6bd8227b35f266a47ae7a5d12788881679524949e71ab8b633d398caf7257d8d"}
Aug 04 18:00:57.939401 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.939007    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:00:57.940160 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.940140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4" event={"ID":"4c173588-249a-419c-850f-d79e5f1ae463","Type":"ContainerStarted","Data":"61397c8c6f6149571abbad8943c23fa5640320f524eba99a084cc73f651f11b9"}
Aug 04 18:00:57.941258 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.941237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d" event={"ID":"ff2119f9-d2f0-44b3-a5e7-71990b0985b1","Type":"ContainerStarted","Data":"f2fe5f8ed02b1c3fc10d9daf78d9a39a0dbbcfd4ccdf00b05b3d4e33d4bb5a1f"}
Aug 04 18:00:57.942286 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.942266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt" event={"ID":"51ab395f-3330-4c91-a828-d2a72c636844","Type":"ContainerStarted","Data":"eb40c5e5b82108a76265da329d2e52873748bb58f91f2f4a64ad4195d2db0e05"}
Aug 04 18:00:57.960561 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:57.960516    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq" podStartSLOduration=0.960502891 podStartE2EDuration="960.502891ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:57.959957609 +0000 UTC m=+65.894027508" watchObservedRunningTime="2026-08-04 18:00:57.960502891 +0000 UTC m=+65.894572768"
Aug 04 18:00:58.251510 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.251442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:58.251510 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.251492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:58.254131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.254112    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:00:58.254231 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.254156    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:00:58.264440 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.264416    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e5dc99-1596-4919-84a4-f8d3bc43ca2d-metrics-certs\") pod \"network-metrics-daemon-x6dkv\" (UID: \"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d\") " pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:58.264529 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.264418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/082c2f9a-47d7-4425-90dc-55ee974b6c58-original-pull-secret\") pod \"global-pull-secret-syncer-glgg2\" (UID: \"082c2f9a-47d7-4425-90dc-55ee974b6c58\") " pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:58.352525 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.352489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:58.352831 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.352814    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-88fc8\""
Aug 04 18:00:58.355211 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.355197    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:00:58.355253 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.355212    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glgg2"
Aug 04 18:00:58.361177 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.361164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x6dkv"
Aug 04 18:00:58.365869 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.365851    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:00:58.376126 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.376106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnb79\" (UniqueName: \"kubernetes.io/projected/87ce351b-62f5-44a3-89a7-ed4a18221c8a-kube-api-access-tnb79\") pod \"network-check-target-rbnmz\" (UID: \"87ce351b-62f5-44a3-89a7-ed4a18221c8a\") " pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:58.499482 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.499406    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-glgg2"]
Aug 04 18:00:58.502956 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:58.502884    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082c2f9a_47d7_4425_90dc_55ee974b6c58.slice/crio-3184474d262c9e8b19e75070adf1df3f8542f9a6ed33f6f98aae3ef2455bf250 WatchSource:0}: Error finding container 3184474d262c9e8b19e75070adf1df3f8542f9a6ed33f6f98aae3ef2455bf250: Status 404 returned error can't find the container with id 3184474d262c9e8b19e75070adf1df3f8542f9a6ed33f6f98aae3ef2455bf250
Aug 04 18:00:58.523881 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.523812    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x6dkv"]
Aug 04 18:00:58.528202 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:58.528169    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e5dc99_1596_4919_84a4_f8d3bc43ca2d.slice/crio-ba4cc3ff13f5f2eece0aeefd6e02195d58c669bf31efdd8149d3bda893e6f233 WatchSource:0}: Error finding container ba4cc3ff13f5f2eece0aeefd6e02195d58c669bf31efdd8149d3bda893e6f233: Status 404 returned error can't find the container with id ba4cc3ff13f5f2eece0aeefd6e02195d58c669bf31efdd8149d3bda893e6f233
Aug 04 18:00:58.663164 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.662914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxtnm\""
Aug 04 18:00:58.670535 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.670240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:00:58.840428 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.840392    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rbnmz"]
Aug 04 18:00:58.845331 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:00:58.845275    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ce351b_62f5_44a3_89a7_ed4a18221c8a.slice/crio-340046df74c39c02276fd489a6e88dff95c67f82f3ab834c23bbca638e337636 WatchSource:0}: Error finding container 340046df74c39c02276fd489a6e88dff95c67f82f3ab834c23bbca638e337636: Status 404 returned error can't find the container with id 340046df74c39c02276fd489a6e88dff95c67f82f3ab834c23bbca638e337636
Aug 04 18:00:58.950380 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.950341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rbnmz" event={"ID":"87ce351b-62f5-44a3-89a7-ed4a18221c8a","Type":"ContainerStarted","Data":"340046df74c39c02276fd489a6e88dff95c67f82f3ab834c23bbca638e337636"}
Aug 04 18:00:58.952685 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.952555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x6dkv" event={"ID":"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d","Type":"ContainerStarted","Data":"ba4cc3ff13f5f2eece0aeefd6e02195d58c669bf31efdd8149d3bda893e6f233"}
Aug 04 18:00:58.954629 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:00:58.954530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-glgg2" event={"ID":"082c2f9a-47d7-4425-90dc-55ee974b6c58","Type":"ContainerStarted","Data":"3184474d262c9e8b19e75070adf1df3f8542f9a6ed33f6f98aae3ef2455bf250"}
Aug 04 18:01:07.064497 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.064464    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:01:07.993174 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.993134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d" event={"ID":"ff2119f9-d2f0-44b3-a5e7-71990b0985b1","Type":"ContainerStarted","Data":"885ec14611888553645aa1fdd689d7260c402c71e023acff690cc762b01e0b3f"}
Aug 04 18:01:07.993380 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.993205    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:01:07.994738 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.994711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-glgg2" event={"ID":"082c2f9a-47d7-4425-90dc-55ee974b6c58","Type":"ContainerStarted","Data":"9e49a6bd0a125cd27c08ef2d9324c5d0fe446e56427ec8c73524504365f23ba4"}
Aug 04 18:01:07.995398 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.995378    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d"
Aug 04 18:01:07.996249 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.996229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt" event={"ID":"51ab395f-3330-4c91-a828-d2a72c636844","Type":"ContainerStarted","Data":"72f2b55e7e283125f4c117898a4132a976b1a1e1aef0de4615029aa709ae50d2"}
Aug 04 18:01:07.998315 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.998284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4" event={"ID":"4c173588-249a-419c-850f-d79e5f1ae463","Type":"ContainerStarted","Data":"aa72d22b46ebdd7cd3ab3b2ded7bd4c700e78cc1afe116822f92167d6a9b49f4"}
Aug 04 18:01:07.998422 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.998320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4" event={"ID":"4c173588-249a-419c-850f-d79e5f1ae463","Type":"ContainerStarted","Data":"79bc1d62ba670aecc703320e45db7ce9b77596e13797b761882f298332875a21"}
Aug 04 18:01:07.998422 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.998337    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4" event={"ID":"4c173588-249a-419c-850f-d79e5f1ae463","Type":"ContainerStarted","Data":"93746f8310894ab75ae06fb0bfb1d7747a67a2e9ea76991e51d9a670cc2f3ca6"}
Aug 04 18:01:07.999839 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.999816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rbnmz" event={"ID":"87ce351b-62f5-44a3-89a7-ed4a18221c8a","Type":"ContainerStarted","Data":"06665abb3a249b0ded3ec591f08119db36bea93c3ef1e1d03342ea0e6617be53"}
Aug 04 18:01:07.999942 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:07.999915    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:01:08.001701 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:08.001679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x6dkv" event={"ID":"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d","Type":"ContainerStarted","Data":"00995216506d03c711401496aac6129b05b70e46b1251e838e9b3d91e0538cf4"}
Aug 04 18:01:08.001786 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:08.001723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x6dkv" event={"ID":"a8e5dc99-1596-4919-84a4-f8d3bc43ca2d","Type":"ContainerStarted","Data":"c66eb07d227583b893dba659ee686c9499a99f1ec9f4b96fcadcb63cb03a0b52"}
Aug 04 18:01:08.033157 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:08.033101    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c4574c57-r5t6d" podStartSLOduration=2.176749484 podStartE2EDuration="12.033080659s" totalImagesPullingTime="9.856331175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.013950162 +0000 UTC m=+75.948020142" watchObservedRunningTime="2026-08-04 18:01:08.033080659 +0000 UTC m=+75.967150560"
Aug 04 18:01:08.050347 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:08.050303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8464fdc88-wbxxt" podStartSLOduration=1.309368251 podStartE2EDuration="11.050285346s" totalImagesPullingTime="9.740917095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.049588941 +0000 UTC m=+75.983658840" watchObservedRunningTime="2026-08-04 18:01:08.050285346 +0000 UTC m=+75.984355245"
Aug 04 18:01:08.071502 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:08.071455    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-glgg2" podStartSLOduration=66.253181915 podStartE2EDuration="1m15.071440249s" totalImagesPullingTime="8.818258334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.070537677 +0000 UTC m=+76.004607571" watchObservedRunningTime="2026-08-04 18:01:08.071440249 +0000 UTC m=+76.005510147"
Aug 04 18:01:08.088464 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:08.088413    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x6dkv" podStartSLOduration=66.289258463 podStartE2EDuration="1m15.088392479s" totalImagesPullingTime="8.799134016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.087800249 +0000 UTC m=+76.021870181" watchObservedRunningTime="2026-08-04 18:01:08.088392479 +0000 UTC m=+76.022462381"
Aug 04 18:01:08.105042 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:08.104995    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rbnmz" podStartSLOduration=66.631104749 podStartE2EDuration="1m15.104981336s" totalImagesPullingTime="8.473876587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.10400234 +0000 UTC m=+76.038072241" watchObservedRunningTime="2026-08-04 18:01:08.104981336 +0000 UTC m=+76.039051234"
Aug 04 18:01:08.125190 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:08.125149    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bcb8b498d-sr9f4" podStartSLOduration=1.312554681 podStartE2EDuration="11.125133837s" totalImagesPullingTime="9.812579156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.123745473 +0000 UTC m=+76.057815384" watchObservedRunningTime="2026-08-04 18:01:08.125133837 +0000 UTC m=+76.059203793"
Aug 04 18:01:16.067249 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.067103    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-97t28"]
Aug 04 18:01:16.071972 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.071949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.074856 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.074761    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 18:01:16.074856 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.074790    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 18:01:16.075024 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.074790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2sv5j\""
Aug 04 18:01:16.075024 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.074902    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 18:01:16.075979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.075955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 18:01:16.076084 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.075987    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 18:01:16.076204 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.076189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 18:01:16.211861 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.211823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwl9\" (UniqueName: \"kubernetes.io/projected/5aa6b949-4a04-48e7-8760-be24399a39eb-kube-api-access-5bwl9\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.212101 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.211879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6b949-4a04-48e7-8760-be24399a39eb-metrics-client-ca\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.212101 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.211923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aa6b949-4a04-48e7-8760-be24399a39eb-sys\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.212101 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.211962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.212101 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.211991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5aa6b949-4a04-48e7-8760-be24399a39eb-root\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.212101 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.212019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-textfile\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.212101 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.212041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-tls\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.212101 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.212072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.212101 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.212099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-wtmp\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313204 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6b949-4a04-48e7-8760-be24399a39eb-metrics-client-ca\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313204 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aa6b949-4a04-48e7-8760-be24399a39eb-sys\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313458 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313458 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5aa6b949-4a04-48e7-8760-be24399a39eb-root\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313458 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-textfile\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313458 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-tls\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313458 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aa6b949-4a04-48e7-8760-be24399a39eb-sys\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313458 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313458 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5aa6b949-4a04-48e7-8760-be24399a39eb-root\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313458 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-wtmp\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313458 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwl9\" (UniqueName: \"kubernetes.io/projected/5aa6b949-4a04-48e7-8760-be24399a39eb-kube-api-access-5bwl9\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313929 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313496    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-wtmp\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.313929 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6b949-4a04-48e7-8760-be24399a39eb-metrics-client-ca\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.314027 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.314027 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.313989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-textfile\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.315722 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.315697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-tls\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.315882 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.315862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5aa6b949-4a04-48e7-8760-be24399a39eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.325285 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.325228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwl9\" (UniqueName: \"kubernetes.io/projected/5aa6b949-4a04-48e7-8760-be24399a39eb-kube-api-access-5bwl9\") pod \"node-exporter-97t28\" (UID: \"5aa6b949-4a04-48e7-8760-be24399a39eb\") " pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.382364 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:16.382322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-97t28"
Aug 04 18:01:16.395096 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:01:16.395028    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa6b949_4a04_48e7_8760_be24399a39eb.slice/crio-3e41d3fc118c627b28b7be39262ac3116b9ff007116a5a7b08ec1054b09742ef WatchSource:0}: Error finding container 3e41d3fc118c627b28b7be39262ac3116b9ff007116a5a7b08ec1054b09742ef: Status 404 returned error can't find the container with id 3e41d3fc118c627b28b7be39262ac3116b9ff007116a5a7b08ec1054b09742ef
Aug 04 18:01:17.037076 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:17.037039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97t28" event={"ID":"5aa6b949-4a04-48e7-8760-be24399a39eb","Type":"ContainerStarted","Data":"3e41d3fc118c627b28b7be39262ac3116b9ff007116a5a7b08ec1054b09742ef"}
Aug 04 18:01:18.041841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:18.041808    2581 generic.go:356] "Generic (PLEG): container finished" podID="5aa6b949-4a04-48e7-8760-be24399a39eb" containerID="d034a55553426be72076a8fb9aa878c8bdd6d70ecff004ea69f0c4457e76c9ee" exitCode=0
Aug 04 18:01:18.042196 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:18.041895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97t28" event={"ID":"5aa6b949-4a04-48e7-8760-be24399a39eb","Type":"ContainerDied","Data":"d034a55553426be72076a8fb9aa878c8bdd6d70ecff004ea69f0c4457e76c9ee"}
Aug 04 18:01:18.959552 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:18.959524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-599d78bb5d-4dgsq"
Aug 04 18:01:19.048565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:19.048527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97t28" event={"ID":"5aa6b949-4a04-48e7-8760-be24399a39eb","Type":"ContainerStarted","Data":"ffd4612a564c0b4864a6e08fe349747cf59824bc4ea6493b561e7e60a0499b38"}
Aug 04 18:01:19.049037 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:19.048592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97t28" event={"ID":"5aa6b949-4a04-48e7-8760-be24399a39eb","Type":"ContainerStarted","Data":"aeb61dcfa18a07b3b6295fc4cbeeeeb10a60bbbc4067cf4fe37da10bb558d276"}
Aug 04 18:01:19.086070 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:19.086020    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-97t28" podStartSLOduration=1.902541062 podStartE2EDuration="3.085999628s" totalImagesPullingTime="1.121426231s" totalInitContainerRuntime="62.032335ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:19.085009159 +0000 UTC m=+87.019079062" watchObservedRunningTime="2026-08-04 18:01:19.085999628 +0000 UTC m=+87.020069528"
Aug 04 18:01:22.078323 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.078285    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f666cc957-mklk8" podUID="d9982f3c-edf0-4beb-a5ed-9f60e24e475d" containerName="registry" containerID="cri-o://86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4" gracePeriod=30
Aug 04 18:01:22.314990 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.314968    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:01:22.356611 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.356519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") pod \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") "
Aug 04 18:01:22.356611 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.356578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-certificates\") pod \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") "
Aug 04 18:01:22.356821 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.356623    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-ca-trust-extracted\") pod \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") "
Aug 04 18:01:22.356821 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.356647    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-bound-sa-token\") pod \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") "
Aug 04 18:01:22.356821 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.356668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnphp\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-kube-api-access-qnphp\") pod \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") "
Aug 04 18:01:22.356821 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.356698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-image-registry-private-configuration\") pod \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") "
Aug 04 18:01:22.356821 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.356724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-installation-pull-secrets\") pod \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") "
Aug 04 18:01:22.356821 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.356783    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-trusted-ca\") pod \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\" (UID: \"d9982f3c-edf0-4beb-a5ed-9f60e24e475d\") "
Aug 04 18:01:22.357108 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.357033    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d9982f3c-edf0-4beb-a5ed-9f60e24e475d" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:22.357524 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.357497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d9982f3c-edf0-4beb-a5ed-9f60e24e475d" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:22.359329 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.359274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d9982f3c-edf0-4beb-a5ed-9f60e24e475d" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:22.359329 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.359289    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d9982f3c-edf0-4beb-a5ed-9f60e24e475d" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:22.359329 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.359310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d9982f3c-edf0-4beb-a5ed-9f60e24e475d" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:22.359474 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.359367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d9982f3c-edf0-4beb-a5ed-9f60e24e475d" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:22.359474 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.359431    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-kube-api-access-qnphp" (OuterVolumeSpecName: "kube-api-access-qnphp") pod "d9982f3c-edf0-4beb-a5ed-9f60e24e475d" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d"). InnerVolumeSpecName "kube-api-access-qnphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:22.365503 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.365481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d9982f3c-edf0-4beb-a5ed-9f60e24e475d" (UID: "d9982f3c-edf0-4beb-a5ed-9f60e24e475d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:01:22.457857 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.457825    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:01:22.457857 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.457853    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-registry-certificates\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:01:22.457857 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.457863    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-ca-trust-extracted\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:01:22.458064 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.457872    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-bound-sa-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:01:22.458064 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.457880    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnphp\" (UniqueName: \"kubernetes.io/projected/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-kube-api-access-qnphp\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:01:22.458064 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.457890    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-image-registry-private-configuration\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:01:22.458064 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.457898    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-installation-pull-secrets\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:01:22.458064 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:22.457907    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9982f3c-edf0-4beb-a5ed-9f60e24e475d-trusted-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:01:23.063883 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:23.063847    2581 generic.go:356] "Generic (PLEG): container finished" podID="d9982f3c-edf0-4beb-a5ed-9f60e24e475d" containerID="86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4" exitCode=0
Aug 04 18:01:23.064050 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:23.063908    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f666cc957-mklk8"
Aug 04 18:01:23.064050 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:23.063936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f666cc957-mklk8" event={"ID":"d9982f3c-edf0-4beb-a5ed-9f60e24e475d","Type":"ContainerDied","Data":"86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4"}
Aug 04 18:01:23.064050 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:23.063991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f666cc957-mklk8" event={"ID":"d9982f3c-edf0-4beb-a5ed-9f60e24e475d","Type":"ContainerDied","Data":"6adccd4daea52ed4765e902b8c341892d9e3a6b4562ce503e2e72c2fbac8e612"}
Aug 04 18:01:23.064050 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:23.064023    2581 scope.go:117] "RemoveContainer" containerID="86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4"
Aug 04 18:01:23.072045 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:23.072029    2581 scope.go:117] "RemoveContainer" containerID="86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4"
Aug 04 18:01:23.072304 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:01:23.072281    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4\": container with ID starting with 86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4 not found: ID does not exist" containerID="86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4"
Aug 04 18:01:23.072365 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:23.072312    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4"} err="failed to get container status \"86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4\": rpc error: code = NotFound desc = could not find container \"86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4\": container with ID starting with 86ae7574c815e515230c4db104f457e5d85b21b9359aae5e8c49f31a988c4dd4 not found: ID does not exist"
Aug 04 18:01:23.114873 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:23.114846    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f666cc957-mklk8"]
Aug 04 18:01:23.121986 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:23.121964    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f666cc957-mklk8"]
Aug 04 18:01:24.545266 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:24.545234    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9982f3c-edf0-4beb-a5ed-9f60e24e475d" path="/var/lib/kubelet/pods/d9982f3c-edf0-4beb-a5ed-9f60e24e475d/volumes"
Aug 04 18:01:39.007904 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:01:39.007865    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rbnmz"
Aug 04 18:04:52.426865 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:04:52.426828    2581 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 18:06:10.846236 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.846159    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dg79b"]
Aug 04 18:06:10.846703 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.846407    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9982f3c-edf0-4beb-a5ed-9f60e24e475d" containerName="registry"
Aug 04 18:06:10.846703 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.846417    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9982f3c-edf0-4beb-a5ed-9f60e24e475d" containerName="registry"
Aug 04 18:06:10.846703 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.846468    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9982f3c-edf0-4beb-a5ed-9f60e24e475d" containerName="registry"
Aug 04 18:06:10.848322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.848307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dg79b"
Aug 04 18:06:10.850879 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.850863    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Aug 04 18:06:10.850954 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.850882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vm7p2\""
Aug 04 18:06:10.852085 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.852055    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Aug 04 18:06:10.856871 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.856846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dg79b"]
Aug 04 18:06:10.979740 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.979697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8a2cb6a-bdc6-4226-aa1a-50be146c07e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-dg79b\" (UID: \"e8a2cb6a-bdc6-4226-aa1a-50be146c07e0\") " pod="cert-manager/cert-manager-55b4d954d5-dg79b"
Aug 04 18:06:10.979915 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:10.979764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gw76\" (UniqueName: \"kubernetes.io/projected/e8a2cb6a-bdc6-4226-aa1a-50be146c07e0-kube-api-access-6gw76\") pod \"cert-manager-55b4d954d5-dg79b\" (UID: \"e8a2cb6a-bdc6-4226-aa1a-50be146c07e0\") " pod="cert-manager/cert-manager-55b4d954d5-dg79b"
Aug 04 18:06:11.080674 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:11.080633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8a2cb6a-bdc6-4226-aa1a-50be146c07e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-dg79b\" (UID: \"e8a2cb6a-bdc6-4226-aa1a-50be146c07e0\") " pod="cert-manager/cert-manager-55b4d954d5-dg79b"
Aug 04 18:06:11.080871 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:11.080698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gw76\" (UniqueName: \"kubernetes.io/projected/e8a2cb6a-bdc6-4226-aa1a-50be146c07e0-kube-api-access-6gw76\") pod \"cert-manager-55b4d954d5-dg79b\" (UID: \"e8a2cb6a-bdc6-4226-aa1a-50be146c07e0\") " pod="cert-manager/cert-manager-55b4d954d5-dg79b"
Aug 04 18:06:11.089166 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:11.089135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8a2cb6a-bdc6-4226-aa1a-50be146c07e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-dg79b\" (UID: \"e8a2cb6a-bdc6-4226-aa1a-50be146c07e0\") " pod="cert-manager/cert-manager-55b4d954d5-dg79b"
Aug 04 18:06:11.089365 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:11.089342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gw76\" (UniqueName: \"kubernetes.io/projected/e8a2cb6a-bdc6-4226-aa1a-50be146c07e0-kube-api-access-6gw76\") pod \"cert-manager-55b4d954d5-dg79b\" (UID: \"e8a2cb6a-bdc6-4226-aa1a-50be146c07e0\") " pod="cert-manager/cert-manager-55b4d954d5-dg79b"
Aug 04 18:06:11.157667 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:11.157619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dg79b"
Aug 04 18:06:11.282450 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:11.282304    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dg79b"]
Aug 04 18:06:11.285096 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:06:11.285069    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a2cb6a_bdc6_4226_aa1a_50be146c07e0.slice/crio-6b1c09ea57708ae222de57793709a66518c5d421094cdbc1782fee1d917ae164 WatchSource:0}: Error finding container 6b1c09ea57708ae222de57793709a66518c5d421094cdbc1782fee1d917ae164: Status 404 returned error can't find the container with id 6b1c09ea57708ae222de57793709a66518c5d421094cdbc1782fee1d917ae164
Aug 04 18:06:11.286863 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:11.286842    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:06:11.994148 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:11.994105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dg79b" event={"ID":"e8a2cb6a-bdc6-4226-aa1a-50be146c07e0","Type":"ContainerStarted","Data":"6b1c09ea57708ae222de57793709a66518c5d421094cdbc1782fee1d917ae164"}
Aug 04 18:06:15.005647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:15.005597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dg79b" event={"ID":"e8a2cb6a-bdc6-4226-aa1a-50be146c07e0","Type":"ContainerStarted","Data":"a31ff8da8cfd6e9193e68ca8e53afae742a492c6c87119ba2a48162744ec97b7"}
Aug 04 18:06:15.024415 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:15.024365    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dg79b" podStartSLOduration=2.15954892 podStartE2EDuration="5.024348322s" totalImagesPullingTime="2.864799402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:15.023763143 +0000 UTC m=+382.957833042" watchObservedRunningTime="2026-08-04 18:06:15.024348322 +0000 UTC m=+382.958418221"
Aug 04 18:06:37.821194 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.821161    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"]
Aug 04 18:06:37.825880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.825857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.828649 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.828623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-pwhz4\""
Aug 04 18:06:37.828771 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.828651    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Aug 04 18:06:37.828771 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.828658    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 04 18:06:37.828771 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.828662    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 04 18:06:37.837640 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.837619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"]
Aug 04 18:06:37.868799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.868768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.868799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.868801    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.868975 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.868831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.868975 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.868884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.868975 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.868911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.868975 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.868931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b38bfa6-6ffe-4039-a812-07860e2747e4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.868975 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.868960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dld47\" (UniqueName: \"kubernetes.io/projected/4b38bfa6-6ffe-4039-a812-07860e2747e4-kube-api-access-dld47\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.869153 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.868985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.869153 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.869032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.969570 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.969570 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.969812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.969812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.969812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.969812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.969812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b38bfa6-6ffe-4039-a812-07860e2747e4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.969812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dld47\" (UniqueName: \"kubernetes.io/projected/4b38bfa6-6ffe-4039-a812-07860e2747e4-kube-api-access-dld47\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.969812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.970163 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.969996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.970163 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.970025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.970163 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.970147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.970427 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.970407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b38bfa6-6ffe-4039-a812-07860e2747e4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.970500 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.970485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.972378 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.972359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.972721 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.972702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.982377 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.982356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b38bfa6-6ffe-4039-a812-07860e2747e4-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:37.985585 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:37.985556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dld47\" (UniqueName: \"kubernetes.io/projected/4b38bfa6-6ffe-4039-a812-07860e2747e4-kube-api-access-dld47\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-d92tq\" (UID: \"4b38bfa6-6ffe-4039-a812-07860e2747e4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:38.136765 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:38.136732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:38.262678 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:38.262662    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"]
Aug 04 18:06:38.264700 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:06:38.264668    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b38bfa6_6ffe_4039_a812_07860e2747e4.slice/crio-2fe09b62161ccde93afc7d815da9c2f312e194d04f9abc7f83849696fea4c19b WatchSource:0}: Error finding container 2fe09b62161ccde93afc7d815da9c2f312e194d04f9abc7f83849696fea4c19b: Status 404 returned error can't find the container with id 2fe09b62161ccde93afc7d815da9c2f312e194d04f9abc7f83849696fea4c19b
Aug 04 18:06:39.092237 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:39.092194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" event={"ID":"4b38bfa6-6ffe-4039-a812-07860e2747e4","Type":"ContainerStarted","Data":"2fe09b62161ccde93afc7d815da9c2f312e194d04f9abc7f83849696fea4c19b"}
Aug 04 18:06:40.924809 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:40.924776    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:06:40.925055 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:40.924849    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:06:40.925055 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:40.924878    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:06:41.100694 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:41.100642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" event={"ID":"4b38bfa6-6ffe-4039-a812-07860e2747e4","Type":"ContainerStarted","Data":"c0151c0e5ffcaff7d1eae59db8390090a3f1f2f2e77451405193c9f043cd6040"}
Aug 04 18:06:41.126588 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:41.126527    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" podStartSLOduration=1.468589145 podStartE2EDuration="4.126506372s" totalImagesPullingTime="2.657917227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:41.125157846 +0000 UTC m=+409.059227746" watchObservedRunningTime="2026-08-04 18:06:41.126506372 +0000 UTC m=+409.060576284"
Aug 04 18:06:41.137645 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:41.137615    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:41.138915 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:41.138892    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Aug 04 18:06:41.139004 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:41.138945    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" podUID="4b38bfa6-6ffe-4039-a812-07860e2747e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Aug 04 18:06:42.140585 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:42.140556    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:43.067935 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.067901    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"]
Aug 04 18:06:43.069945 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.069930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.073867 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.073835    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:06:43.073867 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.073850    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Aug 04 18:06:43.074025 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.073850    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Aug 04 18:06:43.074025 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.073847    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Aug 04 18:06:43.074025 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.073886    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Aug 04 18:06:43.074025 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.073884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pg58q\""
Aug 04 18:06:43.083223 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.083202    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"]
Aug 04 18:06:43.105099 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.105075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m72lv\" (UniqueName: \"kubernetes.io/projected/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-kube-api-access-m72lv\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.105224 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.105138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-manager-config\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.105286 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.105255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-metrics-cert\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.105352 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.105317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-cert\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.107266 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.107248    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:43.108203 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.108189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:06:43.205828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.205799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-cert\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.206254 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.205916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m72lv\" (UniqueName: \"kubernetes.io/projected/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-kube-api-access-m72lv\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.206254 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.205967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-manager-config\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.206254 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.206006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-metrics-cert\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.206821 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.206798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-manager-config\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.208406 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.208378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-cert\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.208652 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.208633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-metrics-cert\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.214755 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.214732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m72lv\" (UniqueName: \"kubernetes.io/projected/3a51f473-6ceb-4ca5-97cc-3bd07f911e53-kube-api-access-m72lv\") pod \"lws-controller-manager-68dff9b49f-gtlwx\" (UID: \"3a51f473-6ceb-4ca5-97cc-3bd07f911e53\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.378980 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.378900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:43.513663 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:43.513634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"]
Aug 04 18:06:43.516203 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:06:43.516176    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a51f473_6ceb_4ca5_97cc_3bd07f911e53.slice/crio-1f664dbddfb4b09b8f14f052e9c1259acf105c41611ed0b1ed6c9e025ed29a7e WatchSource:0}: Error finding container 1f664dbddfb4b09b8f14f052e9c1259acf105c41611ed0b1ed6c9e025ed29a7e: Status 404 returned error can't find the container with id 1f664dbddfb4b09b8f14f052e9c1259acf105c41611ed0b1ed6c9e025ed29a7e
Aug 04 18:06:44.111935 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:44.111895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx" event={"ID":"3a51f473-6ceb-4ca5-97cc-3bd07f911e53","Type":"ContainerStarted","Data":"1f664dbddfb4b09b8f14f052e9c1259acf105c41611ed0b1ed6c9e025ed29a7e"}
Aug 04 18:06:46.121042 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:46.121008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx" event={"ID":"3a51f473-6ceb-4ca5-97cc-3bd07f911e53","Type":"ContainerStarted","Data":"6982e4eb78ab1a3354616084bb96d4127ac4e65afcbf1e5ee099b06ed774b9f5"}
Aug 04 18:06:46.121444 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:46.121136    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:06:46.138955 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:46.138905    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx" podStartSLOduration=0.689923775 podStartE2EDuration="3.138889877s" totalImagesPullingTime="2.448966102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:46.13788684 +0000 UTC m=+414.071956738" watchObservedRunningTime="2026-08-04 18:06:46.138889877 +0000 UTC m=+414.072959776"
Aug 04 18:06:57.127355 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:06:57.127319    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-gtlwx"
Aug 04 18:08:01.471510 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.471473    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-v4675"]
Aug 04 18:08:01.474486 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.474470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:08:01.477543 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.477521    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 04 18:08:01.478831 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.478811    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 04 18:08:01.478932 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.478863    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Aug 04 18:08:01.478932 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.478872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g7sl8\""
Aug 04 18:08:01.494102 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.494080    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-v4675"]
Aug 04 18:08:01.499623 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.499587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hpks\" (UniqueName: \"kubernetes.io/projected/c3ad50db-2758-4dc7-a5a3-a9d9994d64ab-kube-api-access-8hpks\") pod \"limitador-limitador-85d9db4d5d-v4675\" (UID: \"c3ad50db-2758-4dc7-a5a3-a9d9994d64ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:08:01.499738 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.499640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3ad50db-2758-4dc7-a5a3-a9d9994d64ab-config-file\") pod \"limitador-limitador-85d9db4d5d-v4675\" (UID: \"c3ad50db-2758-4dc7-a5a3-a9d9994d64ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:08:01.514418 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.514396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-v4675"]
Aug 04 18:08:01.600236 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.600186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hpks\" (UniqueName: \"kubernetes.io/projected/c3ad50db-2758-4dc7-a5a3-a9d9994d64ab-kube-api-access-8hpks\") pod \"limitador-limitador-85d9db4d5d-v4675\" (UID: \"c3ad50db-2758-4dc7-a5a3-a9d9994d64ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:08:01.600236 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.600242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3ad50db-2758-4dc7-a5a3-a9d9994d64ab-config-file\") pod \"limitador-limitador-85d9db4d5d-v4675\" (UID: \"c3ad50db-2758-4dc7-a5a3-a9d9994d64ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:08:01.600844 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.600826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3ad50db-2758-4dc7-a5a3-a9d9994d64ab-config-file\") pod \"limitador-limitador-85d9db4d5d-v4675\" (UID: \"c3ad50db-2758-4dc7-a5a3-a9d9994d64ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:08:01.609750 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.609732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hpks\" (UniqueName: \"kubernetes.io/projected/c3ad50db-2758-4dc7-a5a3-a9d9994d64ab-kube-api-access-8hpks\") pod \"limitador-limitador-85d9db4d5d-v4675\" (UID: \"c3ad50db-2758-4dc7-a5a3-a9d9994d64ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:08:01.784799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.784706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:08:01.916653 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:01.916631    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-v4675"]
Aug 04 18:08:01.918380 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:08:01.918352    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ad50db_2758_4dc7_a5a3_a9d9994d64ab.slice/crio-d60ba351e5ddb595c075b799a9b63c197902841d72c780a50ee5823795deb946 WatchSource:0}: Error finding container d60ba351e5ddb595c075b799a9b63c197902841d72c780a50ee5823795deb946: Status 404 returned error can't find the container with id d60ba351e5ddb595c075b799a9b63c197902841d72c780a50ee5823795deb946
Aug 04 18:08:02.392349 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:02.392315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675" event={"ID":"c3ad50db-2758-4dc7-a5a3-a9d9994d64ab","Type":"ContainerStarted","Data":"d60ba351e5ddb595c075b799a9b63c197902841d72c780a50ee5823795deb946"}
Aug 04 18:08:04.401520 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:04.401482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675" event={"ID":"c3ad50db-2758-4dc7-a5a3-a9d9994d64ab","Type":"ContainerStarted","Data":"82d098ffee52d88fd1e9f1b3266dc40c7e1255c6fcc8b852f5e80d3fbea99643"}
Aug 04 18:08:04.401967 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:04.401621    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:08:04.418321 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:04.418246    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675" podStartSLOduration=1.252669244 podStartE2EDuration="3.418208728s" totalImagesPullingTime="2.165539484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:04.418025785 +0000 UTC m=+492.352095710" watchObservedRunningTime="2026-08-04 18:08:04.418208728 +0000 UTC m=+492.352278630"
Aug 04 18:08:15.406030 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:08:15.405997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-v4675"
Aug 04 18:10:30.118161 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.118126    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-dxpmm"]
Aug 04 18:10:30.120240 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.120223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dxpmm"
Aug 04 18:10:30.122729 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.122700    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Aug 04 18:10:30.122836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.122732    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 04 18:10:30.122836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.122750    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-drfsn\""
Aug 04 18:10:30.122836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.122818    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 04 18:10:30.129170 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.129152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dxpmm"]
Aug 04 18:10:30.132188 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.132164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e2c34a-24c5-40fd-ace8-ffc777105646-model-cache\") pod \"s3-init-dxpmm\" (UID: \"a3e2c34a-24c5-40fd-ace8-ffc777105646\") " pod="kserve/s3-init-dxpmm"
Aug 04 18:10:30.132284 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.132200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdp8\" (UniqueName: \"kubernetes.io/projected/a3e2c34a-24c5-40fd-ace8-ffc777105646-kube-api-access-wtdp8\") pod \"s3-init-dxpmm\" (UID: \"a3e2c34a-24c5-40fd-ace8-ffc777105646\") " pod="kserve/s3-init-dxpmm"
Aug 04 18:10:30.233442 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.233406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e2c34a-24c5-40fd-ace8-ffc777105646-model-cache\") pod \"s3-init-dxpmm\" (UID: \"a3e2c34a-24c5-40fd-ace8-ffc777105646\") " pod="kserve/s3-init-dxpmm"
Aug 04 18:10:30.233442 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.233438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdp8\" (UniqueName: \"kubernetes.io/projected/a3e2c34a-24c5-40fd-ace8-ffc777105646-kube-api-access-wtdp8\") pod \"s3-init-dxpmm\" (UID: \"a3e2c34a-24c5-40fd-ace8-ffc777105646\") " pod="kserve/s3-init-dxpmm"
Aug 04 18:10:30.233816 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.233795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e2c34a-24c5-40fd-ace8-ffc777105646-model-cache\") pod \"s3-init-dxpmm\" (UID: \"a3e2c34a-24c5-40fd-ace8-ffc777105646\") " pod="kserve/s3-init-dxpmm"
Aug 04 18:10:30.242041 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.242015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdp8\" (UniqueName: \"kubernetes.io/projected/a3e2c34a-24c5-40fd-ace8-ffc777105646-kube-api-access-wtdp8\") pod \"s3-init-dxpmm\" (UID: \"a3e2c34a-24c5-40fd-ace8-ffc777105646\") " pod="kserve/s3-init-dxpmm"
Aug 04 18:10:30.429193 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.429153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dxpmm"
Aug 04 18:10:30.554294 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.554227    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dxpmm"]
Aug 04 18:10:30.556392 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:10:30.556360    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3e2c34a_24c5_40fd_ace8_ffc777105646.slice/crio-55ec320d4a19e4ccb436d7b8fe6999874e03a6c5949dd743fd00f34cca8ed1f9 WatchSource:0}: Error finding container 55ec320d4a19e4ccb436d7b8fe6999874e03a6c5949dd743fd00f34cca8ed1f9: Status 404 returned error can't find the container with id 55ec320d4a19e4ccb436d7b8fe6999874e03a6c5949dd743fd00f34cca8ed1f9
Aug 04 18:10:30.923008 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:30.922977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dxpmm" event={"ID":"a3e2c34a-24c5-40fd-ace8-ffc777105646","Type":"ContainerStarted","Data":"55ec320d4a19e4ccb436d7b8fe6999874e03a6c5949dd743fd00f34cca8ed1f9"}
Aug 04 18:10:34.941065 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:34.941032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dxpmm" event={"ID":"a3e2c34a-24c5-40fd-ace8-ffc777105646","Type":"ContainerStarted","Data":"127a4323375a4a2234e882d56e69918868ba5b1b73af214a47b5c9483d406eab"}
Aug 04 18:10:35.946006 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:35.945966    2581 generic.go:356] "Generic (PLEG): container finished" podID="a3e2c34a-24c5-40fd-ace8-ffc777105646" containerID="127a4323375a4a2234e882d56e69918868ba5b1b73af214a47b5c9483d406eab" exitCode=0
Aug 04 18:10:35.946375 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:35.946039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dxpmm" event={"ID":"a3e2c34a-24c5-40fd-ace8-ffc777105646","Type":"ContainerDied","Data":"127a4323375a4a2234e882d56e69918868ba5b1b73af214a47b5c9483d406eab"}
Aug 04 18:10:40.970687 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:40.970651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dxpmm" event={"ID":"a3e2c34a-24c5-40fd-ace8-ffc777105646","Type":"ContainerStarted","Data":"93bae730b481377e5901fd017323efc594326a286554fe3a018b55a57a1f3c5f"}
Aug 04 18:10:40.989487 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:10:40.989424    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-dxpmm" podStartSLOduration=1.6351152070000001 podStartE2EDuration="10.989406468s" totalImagesPullingTime="7.867537865s" totalInitContainerRuntime="1.486753396s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:40.987319506 +0000 UTC m=+648.921389405" watchObservedRunningTime="2026-08-04 18:10:40.989406468 +0000 UTC m=+648.923476367"
Aug 04 18:11:45.199206 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:45.199171    2581 generic.go:356] "Generic (PLEG): container finished" podID="a3e2c34a-24c5-40fd-ace8-ffc777105646" containerID="93bae730b481377e5901fd017323efc594326a286554fe3a018b55a57a1f3c5f" exitCode=0
Aug 04 18:11:45.199711 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:45.199246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dxpmm" event={"ID":"a3e2c34a-24c5-40fd-ace8-ffc777105646","Type":"ContainerDied","Data":"93bae730b481377e5901fd017323efc594326a286554fe3a018b55a57a1f3c5f"}
Aug 04 18:11:46.326712 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:46.326689    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dxpmm"
Aug 04 18:11:46.431478 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:46.431440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtdp8\" (UniqueName: \"kubernetes.io/projected/a3e2c34a-24c5-40fd-ace8-ffc777105646-kube-api-access-wtdp8\") pod \"a3e2c34a-24c5-40fd-ace8-ffc777105646\" (UID: \"a3e2c34a-24c5-40fd-ace8-ffc777105646\") "
Aug 04 18:11:46.431478 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:46.431483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e2c34a-24c5-40fd-ace8-ffc777105646-model-cache\") pod \"a3e2c34a-24c5-40fd-ace8-ffc777105646\" (UID: \"a3e2c34a-24c5-40fd-ace8-ffc777105646\") "
Aug 04 18:11:46.433784 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:46.433756    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e2c34a-24c5-40fd-ace8-ffc777105646-kube-api-access-wtdp8" (OuterVolumeSpecName: "kube-api-access-wtdp8") pod "a3e2c34a-24c5-40fd-ace8-ffc777105646" (UID: "a3e2c34a-24c5-40fd-ace8-ffc777105646"). InnerVolumeSpecName "kube-api-access-wtdp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:11:46.450232 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:46.450201    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e2c34a-24c5-40fd-ace8-ffc777105646-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3e2c34a-24c5-40fd-ace8-ffc777105646" (UID: "a3e2c34a-24c5-40fd-ace8-ffc777105646"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:11:46.532222 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:46.532145    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e2c34a-24c5-40fd-ace8-ffc777105646-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:11:46.532222 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:46.532179    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtdp8\" (UniqueName: \"kubernetes.io/projected/a3e2c34a-24c5-40fd-ace8-ffc777105646-kube-api-access-wtdp8\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:11:47.207754 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:47.207715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dxpmm" event={"ID":"a3e2c34a-24c5-40fd-ace8-ffc777105646","Type":"ContainerDied","Data":"55ec320d4a19e4ccb436d7b8fe6999874e03a6c5949dd743fd00f34cca8ed1f9"}
Aug 04 18:11:47.207754 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:47.207753    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55ec320d4a19e4ccb436d7b8fe6999874e03a6c5949dd743fd00f34cca8ed1f9"
Aug 04 18:11:47.207754 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:11:47.207722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dxpmm"
Aug 04 18:12:08.350242 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.350204    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 04 18:12:08.350670 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.350481    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e2c34a-24c5-40fd-ace8-ffc777105646" containerName="download-hf-model"
Aug 04 18:12:08.350670 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.350492    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e2c34a-24c5-40fd-ace8-ffc777105646" containerName="download-hf-model"
Aug 04 18:12:08.350670 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.350502    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e2c34a-24c5-40fd-ace8-ffc777105646" containerName="s3-init"
Aug 04 18:12:08.350670 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.350508    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e2c34a-24c5-40fd-ace8-ffc777105646" containerName="s3-init"
Aug 04 18:12:08.350670 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.350563    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e2c34a-24c5-40fd-ace8-ffc777105646" containerName="s3-init"
Aug 04 18:12:08.353496 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.353476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.356452 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.356429    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:12:08.356544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.356433    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Aug 04 18:12:08.356544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.356491    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-vr8jv\""
Aug 04 18:12:08.356655 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.356495    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-9trt8\""
Aug 04 18:12:08.357797 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.357779    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Aug 04 18:12:08.357886 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.357796    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Aug 04 18:12:08.357886 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.357835    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Aug 04 18:12:08.370769 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.370750    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 04 18:12:08.515894 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.515861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.516084 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.515898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.516084 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.515927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f200aca-ceb3-4a83-992a-d93e9800787d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.516084 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.515949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9rj8\" (UniqueName: \"kubernetes.io/projected/8f200aca-ceb3-4a83-992a-d93e9800787d-kube-api-access-g9rj8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.516084 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.516055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.516281 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.516090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.516281 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.516128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.516281 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.516151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f200aca-ceb3-4a83-992a-d93e9800787d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617185 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617185 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617185 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617441 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f200aca-ceb3-4a83-992a-d93e9800787d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617496 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617496 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617625 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617625 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617625 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f200aca-ceb3-4a83-992a-d93e9800787d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617625 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9rj8\" (UniqueName: \"kubernetes.io/projected/8f200aca-ceb3-4a83-992a-d93e9800787d-kube-api-access-g9rj8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.617836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.617794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.618063 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.618045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f200aca-ceb3-4a83-992a-d93e9800787d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.619484 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.619454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.619720 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.619704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f200aca-ceb3-4a83-992a-d93e9800787d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.629923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.629905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9rj8\" (UniqueName: \"kubernetes.io/projected/8f200aca-ceb3-4a83-992a-d93e9800787d-kube-api-access-g9rj8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.663049 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.663028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:08.807148 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:08.807115    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 04 18:12:08.810016 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:12:08.809984    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f200aca_ceb3_4a83_992a_d93e9800787d.slice/crio-3a2c585d416ff870707fb4aad39b9f17c208716454caeabda4060a583d7b425c WatchSource:0}: Error finding container 3a2c585d416ff870707fb4aad39b9f17c208716454caeabda4060a583d7b425c: Status 404 returned error can't find the container with id 3a2c585d416ff870707fb4aad39b9f17c208716454caeabda4060a583d7b425c
Aug 04 18:12:09.287065 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:09.287029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8f200aca-ceb3-4a83-992a-d93e9800787d","Type":"ContainerStarted","Data":"3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9"}
Aug 04 18:12:09.287239 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:09.287081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8f200aca-ceb3-4a83-992a-d93e9800787d","Type":"ContainerStarted","Data":"3a2c585d416ff870707fb4aad39b9f17c208716454caeabda4060a583d7b425c"}
Aug 04 18:12:11.296575 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:11.296482    2581 generic.go:356] "Generic (PLEG): container finished" podID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerID="3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9" exitCode=0
Aug 04 18:12:11.296575 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:11.296536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8f200aca-ceb3-4a83-992a-d93e9800787d","Type":"ContainerDied","Data":"3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9"}
Aug 04 18:12:11.298158 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:11.298134    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:12:19.346835 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:19.346694    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b38bfa6-6ffe-4039-a812-07860e2747e4" containerID="c0151c0e5ffcaff7d1eae59db8390090a3f1f2f2e77451405193c9f043cd6040" exitCode=0
Aug 04 18:12:19.346835 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:19.346765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" event={"ID":"4b38bfa6-6ffe-4039-a812-07860e2747e4","Type":"ContainerDied","Data":"c0151c0e5ffcaff7d1eae59db8390090a3f1f2f2e77451405193c9f043cd6040"}
Aug 04 18:12:19.347373 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:19.347212    2581 scope.go:117] "RemoveContainer" containerID="c0151c0e5ffcaff7d1eae59db8390090a3f1f2f2e77451405193c9f043cd6040"
Aug 04 18:12:19.348418 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:19.348374    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:12:19.348537 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:19.348468    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:12:19.348537 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:19.348525    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:12:20.365647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:20.365590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" event={"ID":"4b38bfa6-6ffe-4039-a812-07860e2747e4","Type":"ContainerStarted","Data":"5e2ea2c1e819f6c039fca3f9c26b007b2ca6c77317b0d8df150bf6b33f599acf"}
Aug 04 18:12:21.150629 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:21.148881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:12:21.150629 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:21.149013    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Aug 04 18:12:21.150629 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:21.149059    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" podUID="4b38bfa6-6ffe-4039-a812-07860e2747e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Aug 04 18:12:22.149634 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:22.149122    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Aug 04 18:12:22.149634 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:22.149203    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" podUID="4b38bfa6-6ffe-4039-a812-07860e2747e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Aug 04 18:12:23.148573 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:23.148118    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Aug 04 18:12:23.148573 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:23.148185    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" podUID="4b38bfa6-6ffe-4039-a812-07860e2747e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Aug 04 18:12:24.149727 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:24.148522    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Aug 04 18:12:24.149727 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:24.148633    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" podUID="4b38bfa6-6ffe-4039-a812-07860e2747e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Aug 04 18:12:25.152476 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:25.152005    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Aug 04 18:12:25.152476 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:25.152081    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq" podUID="4b38bfa6-6ffe-4039-a812-07860e2747e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Aug 04 18:12:26.495697 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:26.494627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:12:26.495697 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:26.495671    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:12:26.497652 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:26.497583    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-d92tq"
Aug 04 18:12:43.465171 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:43.465133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8f200aca-ceb3-4a83-992a-d93e9800787d","Type":"ContainerStarted","Data":"293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef"}
Aug 04 18:12:43.487324 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:43.487272    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=1.731049766 podStartE2EDuration="35.487256432s" totalImagesPullingTime="31.624401508s" totalInitContainerRuntime="2.131805158s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:43.485692557 +0000 UTC m=+771.419762469" watchObservedRunningTime="2026-08-04 18:12:43.487256432 +0000 UTC m=+771.421326331"
Aug 04 18:12:43.663986 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:43.663951    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:48.663548 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:48.663513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:12:48.665089 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:48.665064    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": dial tcp 10.133.0.21:8000: connect: connection refused"
Aug 04 18:12:58.663767 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:12:58.663725    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": dial tcp 10.133.0.21:8000: connect: connection refused"
Aug 04 18:13:08.664100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:13:08.664056    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": dial tcp 10.133.0.21:8000: connect: connection refused"
Aug 04 18:13:18.664723 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:13:18.664627    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": dial tcp 10.133.0.21:8000: connect: connection refused"
Aug 04 18:13:28.664375 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:13:28.664337    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": dial tcp 10.133.0.21:8000: connect: connection refused"
Aug 04 18:13:38.663658 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:13:38.663593    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": dial tcp 10.133.0.21:8000: connect: connection refused"
Aug 04 18:13:48.663890 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:13:48.663848    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": dial tcp 10.133.0.21:8000: connect: connection refused"
Aug 04 18:13:58.664282 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:13:58.664240    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": dial tcp 10.133.0.21:8000: connect: connection refused"
Aug 04 18:14:08.664270 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:08.664224    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": dial tcp 10.133.0.21:8000: connect: connection refused"
Aug 04 18:14:18.673701 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:18.673671    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:14:18.681337 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:18.681310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:14:23.975962 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:23.975928    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 04 18:14:38.997322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:38.997277    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" containerID="cri-o://293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef" gracePeriod=45
Aug 04 18:14:39.752392 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.752368    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:14:39.798517 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f200aca-ceb3-4a83-992a-d93e9800787d-cabundle-cert\") pod \"8f200aca-ceb3-4a83-992a-d93e9800787d\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") "
Aug 04 18:14:39.798517 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798505    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-home\") pod \"8f200aca-ceb3-4a83-992a-d93e9800787d\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") "
Aug 04 18:14:39.798756 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-model-cache\") pod \"8f200aca-ceb3-4a83-992a-d93e9800787d\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") "
Aug 04 18:14:39.798756 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798615    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9rj8\" (UniqueName: \"kubernetes.io/projected/8f200aca-ceb3-4a83-992a-d93e9800787d-kube-api-access-g9rj8\") pod \"8f200aca-ceb3-4a83-992a-d93e9800787d\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") "
Aug 04 18:14:39.798756 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798655    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-kserve-provision-location\") pod \"8f200aca-ceb3-4a83-992a-d93e9800787d\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") "
Aug 04 18:14:39.798756 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f200aca-ceb3-4a83-992a-d93e9800787d-tls-certs\") pod \"8f200aca-ceb3-4a83-992a-d93e9800787d\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") "
Aug 04 18:14:39.798756 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798716    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-dshm\") pod \"8f200aca-ceb3-4a83-992a-d93e9800787d\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") "
Aug 04 18:14:39.798756 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-tmp-dir\") pod \"8f200aca-ceb3-4a83-992a-d93e9800787d\" (UID: \"8f200aca-ceb3-4a83-992a-d93e9800787d\") "
Aug 04 18:14:39.799052 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798847    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f200aca-ceb3-4a83-992a-d93e9800787d" (UID: "8f200aca-ceb3-4a83-992a-d93e9800787d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:39.799052 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.798885    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f200aca-ceb3-4a83-992a-d93e9800787d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f200aca-ceb3-4a83-992a-d93e9800787d" (UID: "8f200aca-ceb3-4a83-992a-d93e9800787d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:39.799052 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.799006    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.799052 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.799023    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f200aca-ceb3-4a83-992a-d93e9800787d-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.799333 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.799292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-home" (OuterVolumeSpecName: "home") pod "8f200aca-ceb3-4a83-992a-d93e9800787d" (UID: "8f200aca-ceb3-4a83-992a-d93e9800787d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:39.801022 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.800999    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-dshm" (OuterVolumeSpecName: "dshm") pod "8f200aca-ceb3-4a83-992a-d93e9800787d" (UID: "8f200aca-ceb3-4a83-992a-d93e9800787d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:39.801150 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.801060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f200aca-ceb3-4a83-992a-d93e9800787d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f200aca-ceb3-4a83-992a-d93e9800787d" (UID: "8f200aca-ceb3-4a83-992a-d93e9800787d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:39.801215 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.801144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f200aca-ceb3-4a83-992a-d93e9800787d-kube-api-access-g9rj8" (OuterVolumeSpecName: "kube-api-access-g9rj8") pod "8f200aca-ceb3-4a83-992a-d93e9800787d" (UID: "8f200aca-ceb3-4a83-992a-d93e9800787d"). InnerVolumeSpecName "kube-api-access-g9rj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:14:39.811576 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.811546    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f200aca-ceb3-4a83-992a-d93e9800787d" (UID: "8f200aca-ceb3-4a83-992a-d93e9800787d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:39.819378 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.819347    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f200aca-ceb3-4a83-992a-d93e9800787d" (UID: "8f200aca-ceb3-4a83-992a-d93e9800787d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:39.899376 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.899340    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.899376 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.899370    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f200aca-ceb3-4a83-992a-d93e9800787d-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.899376 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.899383    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.899619 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.899418    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.899619 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.899428    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f200aca-ceb3-4a83-992a-d93e9800787d-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.899619 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.899439    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9rj8\" (UniqueName: \"kubernetes.io/projected/8f200aca-ceb3-4a83-992a-d93e9800787d-kube-api-access-g9rj8\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.918568 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.918535    2581 generic.go:356] "Generic (PLEG): container finished" podID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerID="293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef" exitCode=0
Aug 04 18:14:39.918722 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.918586    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 04 18:14:39.918722 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.918629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8f200aca-ceb3-4a83-992a-d93e9800787d","Type":"ContainerDied","Data":"293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef"}
Aug 04 18:14:39.918722 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.918679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"8f200aca-ceb3-4a83-992a-d93e9800787d","Type":"ContainerDied","Data":"3a2c585d416ff870707fb4aad39b9f17c208716454caeabda4060a583d7b425c"}
Aug 04 18:14:39.918722 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.918702    2581 scope.go:117] "RemoveContainer" containerID="293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef"
Aug 04 18:14:39.926821 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.926804    2581 scope.go:117] "RemoveContainer" containerID="3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9"
Aug 04 18:14:39.934477 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.934457    2581 scope.go:117] "RemoveContainer" containerID="293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef"
Aug 04 18:14:39.934739 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:14:39.934719    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef\": container with ID starting with 293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef not found: ID does not exist" containerID="293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef"
Aug 04 18:14:39.934787 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.934750    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef"} err="failed to get container status \"293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef\": rpc error: code = NotFound desc = could not find container \"293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef\": container with ID starting with 293b8bb64e7e7e60f690581b64e3d4b6d05209ebb6bbb5ccd4b5ce95ac85cfef not found: ID does not exist"
Aug 04 18:14:39.934787 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.934768    2581 scope.go:117] "RemoveContainer" containerID="3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9"
Aug 04 18:14:39.934986 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:14:39.934972    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9\": container with ID starting with 3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9 not found: ID does not exist" containerID="3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9"
Aug 04 18:14:39.935017 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.934989    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9"} err="failed to get container status \"3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9\": rpc error: code = NotFound desc = could not find container \"3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9\": container with ID starting with 3fd88c69f0b793a9c240c29646ec4a22698bdf40d36b5ea15625f983a03fd3e9 not found: ID does not exist"
Aug 04 18:14:39.939756 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.939735    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 04 18:14:39.945278 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:39.945256    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 04 18:14:40.544672 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:40.544639    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" path="/var/lib/kubelet/pods/8f200aca-ceb3-4a83-992a-d93e9800787d/volumes"
Aug 04 18:14:40.674210 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:40.674166    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": context deadline exceeded"
Aug 04 18:14:58.881488 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.881458    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"]
Aug 04 18:14:58.881810 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.881773    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main"
Aug 04 18:14:58.881810 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.881788    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main"
Aug 04 18:14:58.881810 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.881797    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="storage-initializer"
Aug 04 18:14:58.881810 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.881803    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="storage-initializer"
Aug 04 18:14:58.881940 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.881859    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f200aca-ceb3-4a83-992a-d93e9800787d" containerName="main"
Aug 04 18:14:58.886035 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.886018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:58.889807 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.889785    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Aug 04 18:14:58.890021 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.890006    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-5728g\""
Aug 04 18:14:58.890203 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.890189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Aug 04 18:14:58.891072 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.891055    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Aug 04 18:14:58.899897 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.899873    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"]
Aug 04 18:14:58.943146 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.943109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:58.943326 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.943159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:58.943326 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.943186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:58.943326 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.943228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:58.943326 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.943247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:58.943326 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.943266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:58.943326 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.943296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:58.943326 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.943326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:58.943562 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:58.943361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4zm\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-kube-api-access-lj4zm\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044350 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044350 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044575 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044575 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044575 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044575 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044843 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4zm\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-kube-api-access-lj4zm\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.044979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.045122 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.044977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.045122 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.045007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.045598 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.045571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.045711 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.045677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.046986 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.046963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.047098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.047081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.055956 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.055935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4zm\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-kube-api-access-lj4zm\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.056046 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.055959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-4992v\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.196369 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.196333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:14:59.327903 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.327855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"]
Aug 04 18:14:59.330483 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:14:59.330446    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8baf270_698a_4d7e_b4c1_3fc0062e6704.slice/crio-9198df21a4bfbd2f4e8214e0087e94bd024d99805840d37424de5f58e94ce96e WatchSource:0}: Error finding container 9198df21a4bfbd2f4e8214e0087e94bd024d99805840d37424de5f58e94ce96e: Status 404 returned error can't find the container with id 9198df21a4bfbd2f4e8214e0087e94bd024d99805840d37424de5f58e94ce96e
Aug 04 18:14:59.332389 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.332358    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:14:59.332471 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.332423    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:14:59.332471 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.332451    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:14:59.996771 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.996732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v" event={"ID":"f8baf270-698a-4d7e-b4c1-3fc0062e6704","Type":"ContainerStarted","Data":"3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5"}
Aug 04 18:14:59.997141 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:14:59.996829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v" event={"ID":"f8baf270-698a-4d7e-b4c1-3fc0062e6704","Type":"ContainerStarted","Data":"9198df21a4bfbd2f4e8214e0087e94bd024d99805840d37424de5f58e94ce96e"}
Aug 04 18:15:00.021725 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:00.021679    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v" podStartSLOduration=2.02165923 podStartE2EDuration="2.02165923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:00.0212088 +0000 UTC m=+907.955278700" watchObservedRunningTime="2026-08-04 18:15:00.02165923 +0000 UTC m=+907.955729134"
Aug 04 18:15:00.197149 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:00.197115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:15:00.202003 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:00.201976    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:15:01.000778 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:01.000743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:15:01.001843 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:01.001824    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:15:17.346272 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:17.346236    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"]
Aug 04 18:15:17.346925 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:17.346437    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v" podUID="f8baf270-698a-4d7e-b4c1-3fc0062e6704" containerName="istio-proxy" containerID="cri-o://3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5" gracePeriod=30
Aug 04 18:15:20.384054 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.384030    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:15:20.529726 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.529695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj4zm\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-kube-api-access-lj4zm\") pod \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") "
Aug 04 18:15:20.529888 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.529741    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istiod-ca-cert\") pod \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") "
Aug 04 18:15:20.529888 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.529775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-credential-socket\") pod \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") "
Aug 04 18:15:20.529984 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.529891    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-certs\") pod \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") "
Aug 04 18:15:20.529984 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.529935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-envoy\") pod \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") "
Aug 04 18:15:20.529984 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.529965    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-podinfo\") pod \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") "
Aug 04 18:15:20.530131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.529997    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-data\") pod \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") "
Aug 04 18:15:20.530131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8baf270-698a-4d7e-b4c1-3fc0062e6704" (UID: "f8baf270-698a-4d7e-b4c1-3fc0062e6704"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:20.530131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530061    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-token\") pod \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") "
Aug 04 18:15:20.530131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-socket\") pod \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\" (UID: \"f8baf270-698a-4d7e-b4c1-3fc0062e6704\") "
Aug 04 18:15:20.530335 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8baf270-698a-4d7e-b4c1-3fc0062e6704" (UID: "f8baf270-698a-4d7e-b4c1-3fc0062e6704"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:20.530335 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530201    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8baf270-698a-4d7e-b4c1-3fc0062e6704" (UID: "f8baf270-698a-4d7e-b4c1-3fc0062e6704"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:20.530451 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530360    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8baf270-698a-4d7e-b4c1-3fc0062e6704" (UID: "f8baf270-698a-4d7e-b4c1-3fc0062e6704"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:20.530451 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530377    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8baf270-698a-4d7e-b4c1-3fc0062e6704" (UID: "f8baf270-698a-4d7e-b4c1-3fc0062e6704"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:20.530451 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530407    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:20.530451 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530421    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:20.530451 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.530429    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:20.532161 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.532142    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8baf270-698a-4d7e-b4c1-3fc0062e6704" (UID: "f8baf270-698a-4d7e-b4c1-3fc0062e6704"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:20.532230 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.532214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8baf270-698a-4d7e-b4c1-3fc0062e6704" (UID: "f8baf270-698a-4d7e-b4c1-3fc0062e6704"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:20.532309 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.532291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8baf270-698a-4d7e-b4c1-3fc0062e6704" (UID: "f8baf270-698a-4d7e-b4c1-3fc0062e6704"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 04 18:15:20.532355 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.532322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-kube-api-access-lj4zm" (OuterVolumeSpecName: "kube-api-access-lj4zm") pod "f8baf270-698a-4d7e-b4c1-3fc0062e6704" (UID: "f8baf270-698a-4d7e-b4c1-3fc0062e6704"). InnerVolumeSpecName "kube-api-access-lj4zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:20.631201 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.631171    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:20.631201 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.631193    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:20.631201 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.631205    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:20.631412 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.631214    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:20.631412 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.631221    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8baf270-698a-4d7e-b4c1-3fc0062e6704-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:20.631412 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:20.631230    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4zm\" (UniqueName: \"kubernetes.io/projected/f8baf270-698a-4d7e-b4c1-3fc0062e6704-kube-api-access-lj4zm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:21.084268 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:21.084231    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8baf270-698a-4d7e-b4c1-3fc0062e6704" containerID="3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5" exitCode=0
Aug 04 18:15:21.084443 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:21.084281    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"
Aug 04 18:15:21.084443 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:21.084314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v" event={"ID":"f8baf270-698a-4d7e-b4c1-3fc0062e6704","Type":"ContainerDied","Data":"3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5"}
Aug 04 18:15:21.084443 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:21.084369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v" event={"ID":"f8baf270-698a-4d7e-b4c1-3fc0062e6704","Type":"ContainerDied","Data":"9198df21a4bfbd2f4e8214e0087e94bd024d99805840d37424de5f58e94ce96e"}
Aug 04 18:15:21.084443 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:21.084390    2581 scope.go:117] "RemoveContainer" containerID="3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5"
Aug 04 18:15:21.092211 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:21.092195    2581 scope.go:117] "RemoveContainer" containerID="3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5"
Aug 04 18:15:21.092456 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:15:21.092437    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5\": container with ID starting with 3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5 not found: ID does not exist" containerID="3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5"
Aug 04 18:15:21.092544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:21.092463    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5"} err="failed to get container status \"3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5\": rpc error: code = NotFound desc = could not find container \"3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5\": container with ID starting with 3e8423e3bf624a5abeb6a5615c7d68d331402335c33649aadd509d8dd4d67ec5 not found: ID does not exist"
Aug 04 18:15:21.106927 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:21.106900    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"]
Aug 04 18:15:21.111335 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:21.111315    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-4992v"]
Aug 04 18:15:22.545140 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:22.545105    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8baf270-698a-4d7e-b4c1-3fc0062e6704" path="/var/lib/kubelet/pods/f8baf270-698a-4d7e-b4c1-3fc0062e6704/volumes"
Aug 04 18:15:33.000522 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.000489    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"]
Aug 04 18:15:33.000898 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.000822    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8baf270-698a-4d7e-b4c1-3fc0062e6704" containerName="istio-proxy"
Aug 04 18:15:33.000898 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.000839    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8baf270-698a-4d7e-b4c1-3fc0062e6704" containerName="istio-proxy"
Aug 04 18:15:33.000898 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.000898    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8baf270-698a-4d7e-b4c1-3fc0062e6704" containerName="istio-proxy"
Aug 04 18:15:33.003811 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.003791    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.008091 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.008067    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Aug 04 18:15:33.008205 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.008130    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-mn2ls\""
Aug 04 18:15:33.008205 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.008155    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Aug 04 18:15:33.008327 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.008128    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Aug 04 18:15:33.014819 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.014800    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"]
Aug 04 18:15:33.018831 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.018809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9432777d-cd9a-4342-9105-170e8d9fca89-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.018933 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.018854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.018933 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.018885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.018933 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.018929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.019054 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.019013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.019095 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.019069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55bc\" (UniqueName: \"kubernetes.io/projected/9432777d-cd9a-4342-9105-170e8d9fca89-kube-api-access-l55bc\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.120341 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.120303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9432777d-cd9a-4342-9105-170e8d9fca89-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.120341 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.120348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.120594 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.120369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.120594 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.120407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.120594 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.120436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.120594 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.120476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55bc\" (UniqueName: \"kubernetes.io/projected/9432777d-cd9a-4342-9105-170e8d9fca89-kube-api-access-l55bc\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.120821 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.120806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.120879 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.120844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.120932 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.120887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.122733 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.122717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.122799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.122784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9432777d-cd9a-4342-9105-170e8d9fca89-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.132432 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.132408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55bc\" (UniqueName: \"kubernetes.io/projected/9432777d-cd9a-4342-9105-170e8d9fca89-kube-api-access-l55bc\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.314199 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.314096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:33.437281 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:33.437257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"]
Aug 04 18:15:33.439464 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:15:33.439430    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9432777d_cd9a_4342_9105_170e8d9fca89.slice/crio-865ddd0a80450810ea7dbe9832961cd03f82bf729f5b70c2923ac71fc3628525 WatchSource:0}: Error finding container 865ddd0a80450810ea7dbe9832961cd03f82bf729f5b70c2923ac71fc3628525: Status 404 returned error can't find the container with id 865ddd0a80450810ea7dbe9832961cd03f82bf729f5b70c2923ac71fc3628525
Aug 04 18:15:34.135391 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:34.135355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk" event={"ID":"9432777d-cd9a-4342-9105-170e8d9fca89","Type":"ContainerStarted","Data":"865ddd0a80450810ea7dbe9832961cd03f82bf729f5b70c2923ac71fc3628525"}
Aug 04 18:15:35.140895 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:35.140858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk" event={"ID":"9432777d-cd9a-4342-9105-170e8d9fca89","Type":"ContainerStarted","Data":"875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820"}
Aug 04 18:15:35.160587 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:35.160536    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk" podStartSLOduration=1.7915913190000001 podStartE2EDuration="3.160519578s" totalImagesPullingTime="1.368928259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:35.158417849 +0000 UTC m=+943.092487748" watchObservedRunningTime="2026-08-04 18:15:35.160519578 +0000 UTC m=+943.094589476"
Aug 04 18:15:35.314698 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:35.314664    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:40.510980 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.510942    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"]
Aug 04 18:15:40.519595 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:15:40.519543    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:15:40.519595 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:15:40.519595 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:15:40.519595 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820" cmd=["/bin/sleep","15"]
Aug 04 18:15:40.519886 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.519636    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk" message="" err=<
Aug 04 18:15:40.519886 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:15:40.519886 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:15:40.519886 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:15:40.519886 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:15:40.519657    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:15:40.519886 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:15:40.519886 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:15:40.519886 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk" podUID="9432777d-cd9a-4342-9105-170e8d9fca89" containerName="main" containerID="cri-o://875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820"
Aug 04 18:15:40.519886 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.519713    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk" podUID="9432777d-cd9a-4342-9105-170e8d9fca89" containerName="main" containerID="cri-o://875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820" gracePeriod=60
Aug 04 18:15:40.761213 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.761156    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:40.782640 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.782598    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-tmp-dir\") pod \"9432777d-cd9a-4342-9105-170e8d9fca89\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") "
Aug 04 18:15:40.782767 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.782655    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-dshm\") pod \"9432777d-cd9a-4342-9105-170e8d9fca89\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") "
Aug 04 18:15:40.782767 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.782679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9432777d-cd9a-4342-9105-170e8d9fca89-tls-certs\") pod \"9432777d-cd9a-4342-9105-170e8d9fca89\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") "
Aug 04 18:15:40.782767 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.782702    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-model-cache\") pod \"9432777d-cd9a-4342-9105-170e8d9fca89\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") "
Aug 04 18:15:40.782767 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.782731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-home\") pod \"9432777d-cd9a-4342-9105-170e8d9fca89\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") "
Aug 04 18:15:40.782969 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.782789    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l55bc\" (UniqueName: \"kubernetes.io/projected/9432777d-cd9a-4342-9105-170e8d9fca89-kube-api-access-l55bc\") pod \"9432777d-cd9a-4342-9105-170e8d9fca89\" (UID: \"9432777d-cd9a-4342-9105-170e8d9fca89\") "
Aug 04 18:15:40.782969 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.782848    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9432777d-cd9a-4342-9105-170e8d9fca89" (UID: "9432777d-cd9a-4342-9105-170e8d9fca89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:40.783078 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.782970    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-model-cache" (OuterVolumeSpecName: "model-cache") pod "9432777d-cd9a-4342-9105-170e8d9fca89" (UID: "9432777d-cd9a-4342-9105-170e8d9fca89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:40.783078 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.783036    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-home" (OuterVolumeSpecName: "home") pod "9432777d-cd9a-4342-9105-170e8d9fca89" (UID: "9432777d-cd9a-4342-9105-170e8d9fca89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:40.783078 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.783038    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:40.784835 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.784813    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-dshm" (OuterVolumeSpecName: "dshm") pod "9432777d-cd9a-4342-9105-170e8d9fca89" (UID: "9432777d-cd9a-4342-9105-170e8d9fca89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:40.784835 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.784816    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9432777d-cd9a-4342-9105-170e8d9fca89-kube-api-access-l55bc" (OuterVolumeSpecName: "kube-api-access-l55bc") pod "9432777d-cd9a-4342-9105-170e8d9fca89" (UID: "9432777d-cd9a-4342-9105-170e8d9fca89"). InnerVolumeSpecName "kube-api-access-l55bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:40.784954 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.784892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9432777d-cd9a-4342-9105-170e8d9fca89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9432777d-cd9a-4342-9105-170e8d9fca89" (UID: "9432777d-cd9a-4342-9105-170e8d9fca89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:15:40.884098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.884059    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:40.884098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.884091    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9432777d-cd9a-4342-9105-170e8d9fca89-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:40.884098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.884105    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:40.884329 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.884116    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9432777d-cd9a-4342-9105-170e8d9fca89-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:40.884329 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:40.884127    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55bc\" (UniqueName: \"kubernetes.io/projected/9432777d-cd9a-4342-9105-170e8d9fca89-kube-api-access-l55bc\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:15:41.162951 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:41.162918    2581 generic.go:356] "Generic (PLEG): container finished" podID="9432777d-cd9a-4342-9105-170e8d9fca89" containerID="875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820" exitCode=0
Aug 04 18:15:41.163148 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:41.162959    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"
Aug 04 18:15:41.163148 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:41.162979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk" event={"ID":"9432777d-cd9a-4342-9105-170e8d9fca89","Type":"ContainerDied","Data":"875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820"}
Aug 04 18:15:41.163148 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:41.163031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk" event={"ID":"9432777d-cd9a-4342-9105-170e8d9fca89","Type":"ContainerDied","Data":"865ddd0a80450810ea7dbe9832961cd03f82bf729f5b70c2923ac71fc3628525"}
Aug 04 18:15:41.163148 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:41.163060    2581 scope.go:117] "RemoveContainer" containerID="875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820"
Aug 04 18:15:41.171784 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:41.171767    2581 scope.go:117] "RemoveContainer" containerID="875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820"
Aug 04 18:15:41.172030 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:15:41.172012    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820\": container with ID starting with 875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820 not found: ID does not exist" containerID="875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820"
Aug 04 18:15:41.172078 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:41.172038    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820"} err="failed to get container status \"875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820\": rpc error: code = NotFound desc = could not find container \"875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820\": container with ID starting with 875f94d340aa326d7a4c277d02744fd6786b71c23d71c7c548150126386a1820 not found: ID does not exist"
Aug 04 18:15:41.184791 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:41.184768    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"]
Aug 04 18:15:41.188404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:41.188379    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfhwsxk"]
Aug 04 18:15:42.550038 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:42.550004    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9432777d-cd9a-4342-9105-170e8d9fca89" path="/var/lib/kubelet/pods/9432777d-cd9a-4342-9105-170e8d9fca89/volumes"
Aug 04 18:15:57.509105 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.509067    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"]
Aug 04 18:15:57.509495 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.509474    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9432777d-cd9a-4342-9105-170e8d9fca89" containerName="main"
Aug 04 18:15:57.509495 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.509494    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9432777d-cd9a-4342-9105-170e8d9fca89" containerName="main"
Aug 04 18:15:57.509617 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.509586    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9432777d-cd9a-4342-9105-170e8d9fca89" containerName="main"
Aug 04 18:15:57.514116 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.514083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.517093 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.517067    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Aug 04 18:15:57.517207 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.517090    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Aug 04 18:15:57.517207 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.517143    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Aug 04 18:15:57.517207 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.517171    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Aug 04 18:15:57.517352 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.517318    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-zcs8d\""
Aug 04 18:15:57.517406 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.517353    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:15:57.523864 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.523843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"]
Aug 04 18:15:57.621743 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.621706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-home\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.621941 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.621765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-dshm\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.621941 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.621924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-tmp-dir\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.622057 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.621959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-model-cache\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.622057 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.621993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d3d33b7-4f50-48af-9312-8abc4db4d815-cabundle-cert\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.622133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.622052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3d33b7-4f50-48af-9312-8abc4db4d815-tls-certs\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.622133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.622085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvtgq\" (UniqueName: \"kubernetes.io/projected/2d3d33b7-4f50-48af-9312-8abc4db4d815-kube-api-access-kvtgq\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.622133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.622127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-kserve-provision-location\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.722784 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.722748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-tmp-dir\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.722784 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.722788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-model-cache\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723026 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.722810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d3d33b7-4f50-48af-9312-8abc4db4d815-cabundle-cert\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723026 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.722838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3d33b7-4f50-48af-9312-8abc4db4d815-tls-certs\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723026 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.722862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtgq\" (UniqueName: \"kubernetes.io/projected/2d3d33b7-4f50-48af-9312-8abc4db4d815-kube-api-access-kvtgq\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723026 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.722898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-kserve-provision-location\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723026 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.722950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-home\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723026 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.722974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-dshm\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.723212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-tmp-dir\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.723273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-model-cache\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723405 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.723326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-kserve-provision-location\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723490 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.723466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-home\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.723544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.723527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d3d33b7-4f50-48af-9312-8abc4db4d815-cabundle-cert\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.725161 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.725138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-dshm\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.725338 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.725320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3d33b7-4f50-48af-9312-8abc4db4d815-tls-certs\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.732249 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.732230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtgq\" (UniqueName: \"kubernetes.io/projected/2d3d33b7-4f50-48af-9312-8abc4db4d815-kube-api-access-kvtgq\") pod \"auth-enabled-test-kserve-75548d9f94-6dbzz\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.825703 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.825598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:15:57.957148 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:57.957122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"]
Aug 04 18:15:57.958517 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:15:57.958490    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3d33b7_4f50_48af_9312_8abc4db4d815.slice/crio-f8807080970f6b620fd33c9ad0a946558f01b19de0ebe4a4a6742da7b29113c0 WatchSource:0}: Error finding container f8807080970f6b620fd33c9ad0a946558f01b19de0ebe4a4a6742da7b29113c0: Status 404 returned error can't find the container with id f8807080970f6b620fd33c9ad0a946558f01b19de0ebe4a4a6742da7b29113c0
Aug 04 18:15:58.227014 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:58.226975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" event={"ID":"2d3d33b7-4f50-48af-9312-8abc4db4d815","Type":"ContainerStarted","Data":"6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb"}
Aug 04 18:15:58.227182 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:15:58.227030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" event={"ID":"2d3d33b7-4f50-48af-9312-8abc4db4d815","Type":"ContainerStarted","Data":"f8807080970f6b620fd33c9ad0a946558f01b19de0ebe4a4a6742da7b29113c0"}
Aug 04 18:16:00.235776 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:00.235745    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerID="6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb" exitCode=0
Aug 04 18:16:00.236143 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:00.235805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" event={"ID":"2d3d33b7-4f50-48af-9312-8abc4db4d815","Type":"ContainerDied","Data":"6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb"}
Aug 04 18:16:01.242144 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:01.242109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" event={"ID":"2d3d33b7-4f50-48af-9312-8abc4db4d815","Type":"ContainerStarted","Data":"60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3"}
Aug 04 18:16:01.267735 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:01.267681    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podStartSLOduration=2.257503042 podStartE2EDuration="4.267666564s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.010163522s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:01.265409624 +0000 UTC m=+969.199479523" watchObservedRunningTime="2026-08-04 18:16:01.267666564 +0000 UTC m=+969.201736462"
Aug 04 18:16:01.825871 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:01.825839    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:16:07.826327 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:07.826289    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:16:07.827758 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:07.827730    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:16:17.826881 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:17.826781    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:16:27.826051 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:27.826007    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:16:37.826367 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:37.826322    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:16:47.826910 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:47.826863    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:16:57.826960 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:16:57.826911    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:17:07.826062 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:17:07.826015    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:17:17.826513 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:17:17.826460    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:17:27.826131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:17:27.826077    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:17:37.826023 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:17:37.825969    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:17:47.826382 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:17:47.826295    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Aug 04 18:17:57.836679 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:17:57.836645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:17:57.844617 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:17:57.844573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:18:03.005112 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:03.005074    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"]
Aug 04 18:18:18.026712 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.026649    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main" containerID="cri-o://60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3" gracePeriod=45
Aug 04 18:18:18.769652 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.769627    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:18:18.782404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.782378    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d3d33b7-4f50-48af-9312-8abc4db4d815-cabundle-cert\") pod \"2d3d33b7-4f50-48af-9312-8abc4db4d815\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") "
Aug 04 18:18:18.782509 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.782424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-dshm\") pod \"2d3d33b7-4f50-48af-9312-8abc4db4d815\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") "
Aug 04 18:18:18.782509 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.782448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-kserve-provision-location\") pod \"2d3d33b7-4f50-48af-9312-8abc4db4d815\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") "
Aug 04 18:18:18.782509 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.782476    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3d33b7-4f50-48af-9312-8abc4db4d815-tls-certs\") pod \"2d3d33b7-4f50-48af-9312-8abc4db4d815\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") "
Aug 04 18:18:18.782509 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.782499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-model-cache\") pod \"2d3d33b7-4f50-48af-9312-8abc4db4d815\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") "
Aug 04 18:18:18.782727 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.782529    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-home\") pod \"2d3d33b7-4f50-48af-9312-8abc4db4d815\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") "
Aug 04 18:18:18.782727 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.782551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-tmp-dir\") pod \"2d3d33b7-4f50-48af-9312-8abc4db4d815\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") "
Aug 04 18:18:18.782727 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.782585    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvtgq\" (UniqueName: \"kubernetes.io/projected/2d3d33b7-4f50-48af-9312-8abc4db4d815-kube-api-access-kvtgq\") pod \"2d3d33b7-4f50-48af-9312-8abc4db4d815\" (UID: \"2d3d33b7-4f50-48af-9312-8abc4db4d815\") "
Aug 04 18:18:18.782871 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.782770    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3d33b7-4f50-48af-9312-8abc4db4d815-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d3d33b7-4f50-48af-9312-8abc4db4d815" (UID: "2d3d33b7-4f50-48af-9312-8abc4db4d815"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:18:18.783089 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.783057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d3d33b7-4f50-48af-9312-8abc4db4d815" (UID: "2d3d33b7-4f50-48af-9312-8abc4db4d815"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:18.783523 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.783492    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-home" (OuterVolumeSpecName: "home") pod "2d3d33b7-4f50-48af-9312-8abc4db4d815" (UID: "2d3d33b7-4f50-48af-9312-8abc4db4d815"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:18.785147 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.785109    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3d33b7-4f50-48af-9312-8abc4db4d815-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d3d33b7-4f50-48af-9312-8abc4db4d815" (UID: "2d3d33b7-4f50-48af-9312-8abc4db4d815"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:18:18.785409 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.785352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-dshm" (OuterVolumeSpecName: "dshm") pod "2d3d33b7-4f50-48af-9312-8abc4db4d815" (UID: "2d3d33b7-4f50-48af-9312-8abc4db4d815"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:18.785519 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.785490    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerID="60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3" exitCode=0
Aug 04 18:18:18.785800 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.785521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" event={"ID":"2d3d33b7-4f50-48af-9312-8abc4db4d815","Type":"ContainerDied","Data":"60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3"}
Aug 04 18:18:18.785800 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.785524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3d33b7-4f50-48af-9312-8abc4db4d815-kube-api-access-kvtgq" (OuterVolumeSpecName: "kube-api-access-kvtgq") pod "2d3d33b7-4f50-48af-9312-8abc4db4d815" (UID: "2d3d33b7-4f50-48af-9312-8abc4db4d815"). InnerVolumeSpecName "kube-api-access-kvtgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:18:18.785800 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.785548    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"
Aug 04 18:18:18.785800 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.785574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz" event={"ID":"2d3d33b7-4f50-48af-9312-8abc4db4d815","Type":"ContainerDied","Data":"f8807080970f6b620fd33c9ad0a946558f01b19de0ebe4a4a6742da7b29113c0"}
Aug 04 18:18:18.785800 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.785597    2581 scope.go:117] "RemoveContainer" containerID="60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3"
Aug 04 18:18:18.800436 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.800397    2581 scope.go:117] "RemoveContainer" containerID="6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb"
Aug 04 18:18:18.800708 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.800390    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d3d33b7-4f50-48af-9312-8abc4db4d815" (UID: "2d3d33b7-4f50-48af-9312-8abc4db4d815"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:18.808086 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.808057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d3d33b7-4f50-48af-9312-8abc4db4d815" (UID: "2d3d33b7-4f50-48af-9312-8abc4db4d815"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:18.808495 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.808481    2581 scope.go:117] "RemoveContainer" containerID="60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3"
Aug 04 18:18:18.808796 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:18:18.808768    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3\": container with ID starting with 60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3 not found: ID does not exist" containerID="60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3"
Aug 04 18:18:18.808863 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.808798    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3"} err="failed to get container status \"60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3\": rpc error: code = NotFound desc = could not find container \"60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3\": container with ID starting with 60096ad0f9466f1d055ac98914b651a416a3719224b4f1ae944803d59e8c9de3 not found: ID does not exist"
Aug 04 18:18:18.808863 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.808817    2581 scope.go:117] "RemoveContainer" containerID="6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb"
Aug 04 18:18:18.809050 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:18:18.809032    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb\": container with ID starting with 6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb not found: ID does not exist" containerID="6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb"
Aug 04 18:18:18.809099 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.809055    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb"} err="failed to get container status \"6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb\": rpc error: code = NotFound desc = could not find container \"6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb\": container with ID starting with 6706afb1339fb7337178aa2fd85883e7619677fddf8464b72f57c61ec939cefb not found: ID does not exist"
Aug 04 18:18:18.883660 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.883625    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:18:18.883819 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.883675    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:18:18.883819 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.883687    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvtgq\" (UniqueName: \"kubernetes.io/projected/2d3d33b7-4f50-48af-9312-8abc4db4d815-kube-api-access-kvtgq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:18:18.883819 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.883697    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d3d33b7-4f50-48af-9312-8abc4db4d815-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:18:18.883819 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.883705    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:18:18.883819 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.883713    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:18:18.883819 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.883724    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3d33b7-4f50-48af-9312-8abc4db4d815-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:18:18.883819 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:18.883732    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3d33b7-4f50-48af-9312-8abc4db4d815-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:18:19.112585 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:19.112556    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"]
Aug 04 18:18:19.118557 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:19.118536    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75548d9f94-6dbzz"]
Aug 04 18:18:20.545399 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:18:20.545361    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" path="/var/lib/kubelet/pods/2d3d33b7-4f50-48af-9312-8abc4db4d815/volumes"
Aug 04 18:19:00.687018 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.686989    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"]
Aug 04 18:19:00.687402 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.687244    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="storage-initializer"
Aug 04 18:19:00.687402 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.687255    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="storage-initializer"
Aug 04 18:19:00.687402 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.687272    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main"
Aug 04 18:19:00.687402 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.687278    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main"
Aug 04 18:19:00.687402 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.687322    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d3d33b7-4f50-48af-9312-8abc4db4d815" containerName="main"
Aug 04 18:19:00.691166 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.691146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.694199 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.694176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Aug 04 18:19:00.695201 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.695183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-lhf68\""
Aug 04 18:19:00.695300 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.695184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Aug 04 18:19:00.695300 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.695184    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Aug 04 18:19:00.702682 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.702663    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"]
Aug 04 18:19:00.790532 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.790491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.790775 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.790557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.790775 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.790597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7x99\" (UniqueName: \"kubernetes.io/projected/308629c3-859c-4a98-a755-8c669d46ffec-kube-api-access-f7x99\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.790775 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.790685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.790775 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.790734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/308629c3-859c-4a98-a755-8c669d46ffec-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.790775 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.790758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.891698 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.891653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.891698 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.891708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7x99\" (UniqueName: \"kubernetes.io/projected/308629c3-859c-4a98-a755-8c669d46ffec-kube-api-access-f7x99\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.891953 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.891739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.891953 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.891772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/308629c3-859c-4a98-a755-8c669d46ffec-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.891953 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.891789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.892110 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.892055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.892206 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.892183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.892260 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.892233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.892372 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.892354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.893983 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.893965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.894343 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.894322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/308629c3-859c-4a98-a755-8c669d46ffec-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:00.905505 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:00.905479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7x99\" (UniqueName: \"kubernetes.io/projected/308629c3-859c-4a98-a755-8c669d46ffec-kube-api-access-f7x99\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:01.000389 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:01.000295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:01.129630 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:01.129585    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"]
Aug 04 18:19:01.132049 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:19:01.132024    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308629c3_859c_4a98_a755_8c669d46ffec.slice/crio-e291659b4ae98e17fba0ce86b827b032fce891442cc31b4d5d18cef3f5eb57a1 WatchSource:0}: Error finding container e291659b4ae98e17fba0ce86b827b032fce891442cc31b4d5d18cef3f5eb57a1: Status 404 returned error can't find the container with id e291659b4ae98e17fba0ce86b827b032fce891442cc31b4d5d18cef3f5eb57a1
Aug 04 18:19:01.943860 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:01.943825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8" event={"ID":"308629c3-859c-4a98-a755-8c669d46ffec","Type":"ContainerStarted","Data":"cf4cfdbf3764e1d5025be8519ef2022de29010bc8d0ce401b74633e11c32d758"}
Aug 04 18:19:01.944263 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:01.943874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8" event={"ID":"308629c3-859c-4a98-a755-8c669d46ffec","Type":"ContainerStarted","Data":"e291659b4ae98e17fba0ce86b827b032fce891442cc31b4d5d18cef3f5eb57a1"}
Aug 04 18:19:01.964938 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:01.964890    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8" podStartSLOduration=1.964875494 podStartE2EDuration="1.964875494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:01.963892214 +0000 UTC m=+1149.897962113" watchObservedRunningTime="2026-08-04 18:19:01.964875494 +0000 UTC m=+1149.898945393"
Aug 04 18:19:02.000756 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:02.000725    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:11.001457 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:11.001416    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:11.019357 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:11.019329    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:11.989995 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:11.989967    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:34.508655 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:34.508624    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"]
Aug 04 18:19:34.517281 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:19:34.517244    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:19:34.517281 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:19:34.517281 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:19:34.517281 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="cf4cfdbf3764e1d5025be8519ef2022de29010bc8d0ce401b74633e11c32d758" cmd=["/bin/sleep","15"]
Aug 04 18:19:34.517527 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:34.517295    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8" message="" err=<
Aug 04 18:19:34.517527 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:19:34.517527 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:19:34.517527 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:19:34.517527 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:19:34.517307    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:19:34.517527 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:19:34.517527 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:19:34.517527 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8" podUID="308629c3-859c-4a98-a755-8c669d46ffec" containerName="main" containerID="cri-o://cf4cfdbf3764e1d5025be8519ef2022de29010bc8d0ce401b74633e11c32d758"
Aug 04 18:19:34.517527 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:34.517358    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8" podUID="308629c3-859c-4a98-a755-8c669d46ffec" containerName="main" containerID="cri-o://cf4cfdbf3764e1d5025be8519ef2022de29010bc8d0ce401b74633e11c32d758" gracePeriod=60
Aug 04 18:19:34.979206 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:34.979160    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8" podUID="308629c3-859c-4a98-a755-8c669d46ffec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/health\": dial tcp 10.133.0.25:8000: connect: connection refused"
Aug 04 18:19:35.066344 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.066308    2581 generic.go:356] "Generic (PLEG): container finished" podID="308629c3-859c-4a98-a755-8c669d46ffec" containerID="cf4cfdbf3764e1d5025be8519ef2022de29010bc8d0ce401b74633e11c32d758" exitCode=0
Aug 04 18:19:35.066507 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.066364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8" event={"ID":"308629c3-859c-4a98-a755-8c669d46ffec","Type":"ContainerDied","Data":"cf4cfdbf3764e1d5025be8519ef2022de29010bc8d0ce401b74633e11c32d758"}
Aug 04 18:19:35.264336 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.264310    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:35.269534 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.269508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/308629c3-859c-4a98-a755-8c669d46ffec-tls-certs\") pod \"308629c3-859c-4a98-a755-8c669d46ffec\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") "
Aug 04 18:19:35.269645 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.269538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-tmp-dir\") pod \"308629c3-859c-4a98-a755-8c669d46ffec\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") "
Aug 04 18:19:35.269645 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.269558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-home\") pod \"308629c3-859c-4a98-a755-8c669d46ffec\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") "
Aug 04 18:19:35.269645 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.269596    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-dshm\") pod \"308629c3-859c-4a98-a755-8c669d46ffec\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") "
Aug 04 18:19:35.269783 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.269649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-model-cache\") pod \"308629c3-859c-4a98-a755-8c669d46ffec\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") "
Aug 04 18:19:35.269783 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.269690    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7x99\" (UniqueName: \"kubernetes.io/projected/308629c3-859c-4a98-a755-8c669d46ffec-kube-api-access-f7x99\") pod \"308629c3-859c-4a98-a755-8c669d46ffec\" (UID: \"308629c3-859c-4a98-a755-8c669d46ffec\") "
Aug 04 18:19:35.269883 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.269815    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "308629c3-859c-4a98-a755-8c669d46ffec" (UID: "308629c3-859c-4a98-a755-8c669d46ffec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:19:35.269883 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.269844    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-home" (OuterVolumeSpecName: "home") pod "308629c3-859c-4a98-a755-8c669d46ffec" (UID: "308629c3-859c-4a98-a755-8c669d46ffec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:19:35.269987 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.269947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-model-cache" (OuterVolumeSpecName: "model-cache") pod "308629c3-859c-4a98-a755-8c669d46ffec" (UID: "308629c3-859c-4a98-a755-8c669d46ffec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:19:35.271648 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.271574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/308629c3-859c-4a98-a755-8c669d46ffec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "308629c3-859c-4a98-a755-8c669d46ffec" (UID: "308629c3-859c-4a98-a755-8c669d46ffec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:19:35.271729 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.271702    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-dshm" (OuterVolumeSpecName: "dshm") pod "308629c3-859c-4a98-a755-8c669d46ffec" (UID: "308629c3-859c-4a98-a755-8c669d46ffec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:19:35.271778 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.271755    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308629c3-859c-4a98-a755-8c669d46ffec-kube-api-access-f7x99" (OuterVolumeSpecName: "kube-api-access-f7x99") pod "308629c3-859c-4a98-a755-8c669d46ffec" (UID: "308629c3-859c-4a98-a755-8c669d46ffec"). InnerVolumeSpecName "kube-api-access-f7x99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:19:35.370916 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.370886    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/308629c3-859c-4a98-a755-8c669d46ffec-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:19:35.370916 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.370910    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:19:35.370916 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.370918    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:19:35.370916 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.370925    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:19:35.371165 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.370933    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/308629c3-859c-4a98-a755-8c669d46ffec-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:19:35.371165 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:35.370941    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7x99\" (UniqueName: \"kubernetes.io/projected/308629c3-859c-4a98-a755-8c669d46ffec-kube-api-access-f7x99\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:19:36.071702 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:36.071671    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"
Aug 04 18:19:36.071702 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:36.071683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8" event={"ID":"308629c3-859c-4a98-a755-8c669d46ffec","Type":"ContainerDied","Data":"e291659b4ae98e17fba0ce86b827b032fce891442cc31b4d5d18cef3f5eb57a1"}
Aug 04 18:19:36.072251 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:36.071760    2581 scope.go:117] "RemoveContainer" containerID="cf4cfdbf3764e1d5025be8519ef2022de29010bc8d0ce401b74633e11c32d758"
Aug 04 18:19:36.093694 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:36.093667    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"]
Aug 04 18:19:36.098338 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:36.098317    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jj2j8"]
Aug 04 18:19:36.545757 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:36.545724    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="308629c3-859c-4a98-a755-8c669d46ffec" path="/var/lib/kubelet/pods/308629c3-859c-4a98-a755-8c669d46ffec/volumes"
Aug 04 18:19:57.699969 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.699938    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"]
Aug 04 18:19:57.700446 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.700204    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308629c3-859c-4a98-a755-8c669d46ffec" containerName="main"
Aug 04 18:19:57.700446 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.700213    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="308629c3-859c-4a98-a755-8c669d46ffec" containerName="main"
Aug 04 18:19:57.700446 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.700278    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="308629c3-859c-4a98-a755-8c669d46ffec" containerName="main"
Aug 04 18:19:57.703583 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.703558    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.706555 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.706536    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Aug 04 18:19:57.706681 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.706539    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Aug 04 18:19:57.708339 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.708318    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-rflsd\""
Aug 04 18:19:57.708429 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.708320    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Aug 04 18:19:57.711085 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.711056    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"]
Aug 04 18:19:57.742901 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.742873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.743055 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.742915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hb48\" (UniqueName: \"kubernetes.io/projected/09346319-c549-489d-be11-bab7b44e8c9a-kube-api-access-4hb48\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.743055 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.742972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.743055 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.742990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.743055 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.743008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09346319-c549-489d-be11-bab7b44e8c9a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.743192 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.743068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.843778 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.843749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hb48\" (UniqueName: \"kubernetes.io/projected/09346319-c549-489d-be11-bab7b44e8c9a-kube-api-access-4hb48\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.843995 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.843806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.843995 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.843916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.843995 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.843960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09346319-c549-489d-be11-bab7b44e8c9a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.844295 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.844008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.844295 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.844039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.844434 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.844315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.844434 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.844327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.844434 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.844373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.845986 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.845966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.846318 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.846301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09346319-c549-489d-be11-bab7b44e8c9a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:57.852183 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:57.852164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hb48\" (UniqueName: \"kubernetes.io/projected/09346319-c549-489d-be11-bab7b44e8c9a-kube-api-access-4hb48\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ccg5n\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:58.014884 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:58.014794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:19:58.142475 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:58.142435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"]
Aug 04 18:19:58.145360 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:19:58.145332    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09346319_c549_489d_be11_bab7b44e8c9a.slice/crio-40a3a00ca4f8df8215ea3e6f5faa1e2efda6b704126e24f64a0aecc2515bc49f WatchSource:0}: Error finding container 40a3a00ca4f8df8215ea3e6f5faa1e2efda6b704126e24f64a0aecc2515bc49f: Status 404 returned error can't find the container with id 40a3a00ca4f8df8215ea3e6f5faa1e2efda6b704126e24f64a0aecc2515bc49f
Aug 04 18:19:58.155339 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:58.155316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n" event={"ID":"09346319-c549-489d-be11-bab7b44e8c9a","Type":"ContainerStarted","Data":"40a3a00ca4f8df8215ea3e6f5faa1e2efda6b704126e24f64a0aecc2515bc49f"}
Aug 04 18:19:59.160100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:59.160059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n" event={"ID":"09346319-c549-489d-be11-bab7b44e8c9a","Type":"ContainerStarted","Data":"4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32"}
Aug 04 18:19:59.178710 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:19:59.178664    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n" podStartSLOduration=2.178647985 podStartE2EDuration="2.178647985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:59.177198921 +0000 UTC m=+1207.111268841" watchObservedRunningTime="2026-08-04 18:19:59.178647985 +0000 UTC m=+1207.112717883"
Aug 04 18:20:00.015150 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:00.015119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:20:08.015746 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:08.015707    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:20:08.027580 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:08.027554    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:20:08.205069 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:08.205042    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:20:31.377314 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.377269    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"]
Aug 04 18:20:31.386097 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:20:31.386063    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:20:31.386097 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:20:31.386097 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:20:31.386097 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32" cmd=["/bin/sleep","15"]
Aug 04 18:20:31.386316 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.386122    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n" message="" err=<
Aug 04 18:20:31.386316 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:20:31.386316 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:20:31.386316 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:20:31.386316 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:20:31.386141    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:20:31.386316 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:20:31.386316 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:20:31.386316 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n" podUID="09346319-c549-489d-be11-bab7b44e8c9a" containerName="main" containerID="cri-o://4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32"
Aug 04 18:20:31.386316 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.386194    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n" podUID="09346319-c549-489d-be11-bab7b44e8c9a" containerName="main" containerID="cri-o://4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32" gracePeriod=60
Aug 04 18:20:31.730828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.730801    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:20:31.818776 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.818749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-dshm\") pod \"09346319-c549-489d-be11-bab7b44e8c9a\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") "
Aug 04 18:20:31.818956 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.818792    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-model-cache\") pod \"09346319-c549-489d-be11-bab7b44e8c9a\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") "
Aug 04 18:20:31.818956 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.818854    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-home\") pod \"09346319-c549-489d-be11-bab7b44e8c9a\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") "
Aug 04 18:20:31.818956 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.818880    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-tmp-dir\") pod \"09346319-c549-489d-be11-bab7b44e8c9a\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") "
Aug 04 18:20:31.818956 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.818912    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09346319-c549-489d-be11-bab7b44e8c9a-tls-certs\") pod \"09346319-c549-489d-be11-bab7b44e8c9a\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") "
Aug 04 18:20:31.818956 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.818943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hb48\" (UniqueName: \"kubernetes.io/projected/09346319-c549-489d-be11-bab7b44e8c9a-kube-api-access-4hb48\") pod \"09346319-c549-489d-be11-bab7b44e8c9a\" (UID: \"09346319-c549-489d-be11-bab7b44e8c9a\") "
Aug 04 18:20:31.819229 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.819138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-home" (OuterVolumeSpecName: "home") pod "09346319-c549-489d-be11-bab7b44e8c9a" (UID: "09346319-c549-489d-be11-bab7b44e8c9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:31.819229 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.819150    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "09346319-c549-489d-be11-bab7b44e8c9a" (UID: "09346319-c549-489d-be11-bab7b44e8c9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:31.819229 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.819169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09346319-c549-489d-be11-bab7b44e8c9a" (UID: "09346319-c549-489d-be11-bab7b44e8c9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:31.821121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.821091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09346319-c549-489d-be11-bab7b44e8c9a-kube-api-access-4hb48" (OuterVolumeSpecName: "kube-api-access-4hb48") pod "09346319-c549-489d-be11-bab7b44e8c9a" (UID: "09346319-c549-489d-be11-bab7b44e8c9a"). InnerVolumeSpecName "kube-api-access-4hb48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:20:31.821243 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.821140    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-dshm" (OuterVolumeSpecName: "dshm") pod "09346319-c549-489d-be11-bab7b44e8c9a" (UID: "09346319-c549-489d-be11-bab7b44e8c9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:31.821243 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.821151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09346319-c549-489d-be11-bab7b44e8c9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09346319-c549-489d-be11-bab7b44e8c9a" (UID: "09346319-c549-489d-be11-bab7b44e8c9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:20:31.920133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.920031    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:20:31.920133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.920059    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:20:31.920133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.920069    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09346319-c549-489d-be11-bab7b44e8c9a-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:20:31.920133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.920077    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hb48\" (UniqueName: \"kubernetes.io/projected/09346319-c549-489d-be11-bab7b44e8c9a-kube-api-access-4hb48\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:20:31.920133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.920086    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:20:31.920133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:31.920094    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09346319-c549-489d-be11-bab7b44e8c9a-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:20:32.288772 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.288685    2581 generic.go:356] "Generic (PLEG): container finished" podID="09346319-c549-489d-be11-bab7b44e8c9a" containerID="4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32" exitCode=0
Aug 04 18:20:32.288772 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.288738    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"
Aug 04 18:20:32.288942 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.288773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n" event={"ID":"09346319-c549-489d-be11-bab7b44e8c9a","Type":"ContainerDied","Data":"4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32"}
Aug 04 18:20:32.288942 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.288835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n" event={"ID":"09346319-c549-489d-be11-bab7b44e8c9a","Type":"ContainerDied","Data":"40a3a00ca4f8df8215ea3e6f5faa1e2efda6b704126e24f64a0aecc2515bc49f"}
Aug 04 18:20:32.288942 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.288859    2581 scope.go:117] "RemoveContainer" containerID="4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32"
Aug 04 18:20:32.297220 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.297204    2581 scope.go:117] "RemoveContainer" containerID="4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32"
Aug 04 18:20:32.297446 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:20:32.297430    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32\": container with ID starting with 4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32 not found: ID does not exist" containerID="4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32"
Aug 04 18:20:32.297533 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.297452    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32"} err="failed to get container status \"4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32\": rpc error: code = NotFound desc = could not find container \"4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32\": container with ID starting with 4ad75dcc71b6a6919b3b8e1df29228e20b68af9528c1de81d336c6cdf45c8a32 not found: ID does not exist"
Aug 04 18:20:32.309741 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.309718    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"]
Aug 04 18:20:32.313265 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.313243    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ccg5n"]
Aug 04 18:20:32.545416 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:20:32.545338    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09346319-c549-489d-be11-bab7b44e8c9a" path="/var/lib/kubelet/pods/09346319-c549-489d-be11-bab7b44e8c9a/volumes"
Aug 04 18:21:04.853189 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.853157    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"]
Aug 04 18:21:04.853705 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.853625    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09346319-c549-489d-be11-bab7b44e8c9a" containerName="main"
Aug 04 18:21:04.853705 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.853658    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="09346319-c549-489d-be11-bab7b44e8c9a" containerName="main"
Aug 04 18:21:04.853832 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.853731    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="09346319-c549-489d-be11-bab7b44e8c9a" containerName="main"
Aug 04 18:21:04.857690 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.857670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:04.860494 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.860474    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Aug 04 18:21:04.860494 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.860488    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Aug 04 18:21:04.861548 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.861530    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-xdbl6\""
Aug 04 18:21:04.861827 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.861813    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Aug 04 18:21:04.868322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.868304    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"]
Aug 04 18:21:04.972551 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.972512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:04.972551 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.972549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:04.972763 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.972632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:04.972763 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.972670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923ce662-c8ab-44dc-8079-1543f5f476ea-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:04.972763 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.972692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjdz\" (UniqueName: \"kubernetes.io/projected/923ce662-c8ab-44dc-8079-1543f5f476ea-kube-api-access-9hjdz\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:04.972861 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:04.972771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.073522 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.073468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.073739 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.073537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.073739 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.073568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.073739 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.073631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.073739 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.073658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923ce662-c8ab-44dc-8079-1543f5f476ea-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.073739 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.073683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjdz\" (UniqueName: \"kubernetes.io/projected/923ce662-c8ab-44dc-8079-1543f5f476ea-kube-api-access-9hjdz\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.073996 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.073879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.073996 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.073959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.074087 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.074052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.075867 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.075844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.076149 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.076130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923ce662-c8ab-44dc-8079-1543f5f476ea-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.082823 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.082803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjdz\" (UniqueName: \"kubernetes.io/projected/923ce662-c8ab-44dc-8079-1543f5f476ea-kube-api-access-9hjdz\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.167568 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.167541    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:05.296163 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.296138    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"]
Aug 04 18:21:05.298550 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:21:05.298523    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923ce662_c8ab_44dc_8079_1543f5f476ea.slice/crio-bcbbc019ff04efbca5bd381fcdd0888912c65d96da900e023e56a868a548b80a WatchSource:0}: Error finding container bcbbc019ff04efbca5bd381fcdd0888912c65d96da900e023e56a868a548b80a: Status 404 returned error can't find the container with id bcbbc019ff04efbca5bd381fcdd0888912c65d96da900e023e56a868a548b80a
Aug 04 18:21:05.412559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.412521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m" event={"ID":"923ce662-c8ab-44dc-8079-1543f5f476ea","Type":"ContainerStarted","Data":"bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2"}
Aug 04 18:21:05.412751 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.412580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m" event={"ID":"923ce662-c8ab-44dc-8079-1543f5f476ea","Type":"ContainerStarted","Data":"bcbbc019ff04efbca5bd381fcdd0888912c65d96da900e023e56a868a548b80a"}
Aug 04 18:21:05.430842 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:05.430736    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m" podStartSLOduration=1.430718066 podStartE2EDuration="1.430718066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:05.429421087 +0000 UTC m=+1273.363490986" watchObservedRunningTime="2026-08-04 18:21:05.430718066 +0000 UTC m=+1273.364787969"
Aug 04 18:21:06.168117 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:06.168087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:15.168244 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:15.168211    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:15.180746 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:15.180725    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:21:15.460246 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:21:15.460165    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:22:43.176546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.176509    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"]
Aug 04 18:22:43.184883 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:22:43.184846    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:22:43.184883 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:22:43.184883 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:22:43.184883 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2" cmd=["/bin/sleep","15"]
Aug 04 18:22:43.185106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.184900    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m" message="" err=<
Aug 04 18:22:43.185106 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:22:43.185106 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:22:43.185106 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:22:43.185106 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:22:43.184912    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:22:43.185106 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:22:43.185106 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:22:43.185106 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m" podUID="923ce662-c8ab-44dc-8079-1543f5f476ea" containerName="main" containerID="cri-o://bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2"
Aug 04 18:22:43.185106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.184954    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m" podUID="923ce662-c8ab-44dc-8079-1543f5f476ea" containerName="main" containerID="cri-o://bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2" gracePeriod=60
Aug 04 18:22:43.449780 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.449687    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m" podUID="923ce662-c8ab-44dc-8079-1543f5f476ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/health\": dial tcp 10.133.0.27:8000: connect: connection refused"
Aug 04 18:22:43.523931 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.523903    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:22:43.560578 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.560552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-model-cache\") pod \"923ce662-c8ab-44dc-8079-1543f5f476ea\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") "
Aug 04 18:22:43.560734 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.560583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923ce662-c8ab-44dc-8079-1543f5f476ea-tls-certs\") pod \"923ce662-c8ab-44dc-8079-1543f5f476ea\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") "
Aug 04 18:22:43.560734 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.560625    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-dshm\") pod \"923ce662-c8ab-44dc-8079-1543f5f476ea\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") "
Aug 04 18:22:43.560734 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.560674    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-home\") pod \"923ce662-c8ab-44dc-8079-1543f5f476ea\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") "
Aug 04 18:22:43.560734 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.560715    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hjdz\" (UniqueName: \"kubernetes.io/projected/923ce662-c8ab-44dc-8079-1543f5f476ea-kube-api-access-9hjdz\") pod \"923ce662-c8ab-44dc-8079-1543f5f476ea\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") "
Aug 04 18:22:43.560924 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.560761    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-tmp-dir\") pod \"923ce662-c8ab-44dc-8079-1543f5f476ea\" (UID: \"923ce662-c8ab-44dc-8079-1543f5f476ea\") "
Aug 04 18:22:43.560924 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.560867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "923ce662-c8ab-44dc-8079-1543f5f476ea" (UID: "923ce662-c8ab-44dc-8079-1543f5f476ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:43.561036 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.560964    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.561090 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.561047    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "923ce662-c8ab-44dc-8079-1543f5f476ea" (UID: "923ce662-c8ab-44dc-8079-1543f5f476ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:43.561090 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.561069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-home" (OuterVolumeSpecName: "home") pod "923ce662-c8ab-44dc-8079-1543f5f476ea" (UID: "923ce662-c8ab-44dc-8079-1543f5f476ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:43.562952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.562927    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-dshm" (OuterVolumeSpecName: "dshm") pod "923ce662-c8ab-44dc-8079-1543f5f476ea" (UID: "923ce662-c8ab-44dc-8079-1543f5f476ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:43.563067 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.563046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923ce662-c8ab-44dc-8079-1543f5f476ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "923ce662-c8ab-44dc-8079-1543f5f476ea" (UID: "923ce662-c8ab-44dc-8079-1543f5f476ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:22:43.563127 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.563046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923ce662-c8ab-44dc-8079-1543f5f476ea-kube-api-access-9hjdz" (OuterVolumeSpecName: "kube-api-access-9hjdz") pod "923ce662-c8ab-44dc-8079-1543f5f476ea" (UID: "923ce662-c8ab-44dc-8079-1543f5f476ea"). InnerVolumeSpecName "kube-api-access-9hjdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:22:43.662209 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.662160    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.662209 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.662201    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923ce662-c8ab-44dc-8079-1543f5f476ea-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.662209 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.662210    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.662209 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.662218    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923ce662-c8ab-44dc-8079-1543f5f476ea-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.662209 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.662226    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hjdz\" (UniqueName: \"kubernetes.io/projected/923ce662-c8ab-44dc-8079-1543f5f476ea-kube-api-access-9hjdz\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.779180 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.779089    2581 generic.go:356] "Generic (PLEG): container finished" podID="923ce662-c8ab-44dc-8079-1543f5f476ea" containerID="bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2" exitCode=0
Aug 04 18:22:43.779180 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.779137    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"
Aug 04 18:22:43.779180 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.779156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m" event={"ID":"923ce662-c8ab-44dc-8079-1543f5f476ea","Type":"ContainerDied","Data":"bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2"}
Aug 04 18:22:43.779419 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.779197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m" event={"ID":"923ce662-c8ab-44dc-8079-1543f5f476ea","Type":"ContainerDied","Data":"bcbbc019ff04efbca5bd381fcdd0888912c65d96da900e023e56a868a548b80a"}
Aug 04 18:22:43.779419 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.779221    2581 scope.go:117] "RemoveContainer" containerID="bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2"
Aug 04 18:22:43.787448 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.787428    2581 scope.go:117] "RemoveContainer" containerID="bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2"
Aug 04 18:22:43.787749 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:22:43.787732    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2\": container with ID starting with bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2 not found: ID does not exist" containerID="bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2"
Aug 04 18:22:43.787799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.787759    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2"} err="failed to get container status \"bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2\": rpc error: code = NotFound desc = could not find container \"bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2\": container with ID starting with bc23749c5a2046ea6809cf6499a50ff171b4062f4f09c751e480bcd88dccc1e2 not found: ID does not exist"
Aug 04 18:22:43.802558 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.802533    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"]
Aug 04 18:22:43.807328 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:43.807305    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvm4m"]
Aug 04 18:22:44.545054 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:22:44.545020    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="923ce662-c8ab-44dc-8079-1543f5f476ea" path="/var/lib/kubelet/pods/923ce662-c8ab-44dc-8079-1543f5f476ea/volumes"
Aug 04 18:23:17.450333 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.450295    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"]
Aug 04 18:23:17.450811 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.450598    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="923ce662-c8ab-44dc-8079-1543f5f476ea" containerName="main"
Aug 04 18:23:17.450811 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.450625    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="923ce662-c8ab-44dc-8079-1543f5f476ea" containerName="main"
Aug 04 18:23:17.450811 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.450674    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="923ce662-c8ab-44dc-8079-1543f5f476ea" containerName="main"
Aug 04 18:23:17.455017 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.454992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.458965 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.458940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-pt5nt\""
Aug 04 18:23:17.459142 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.458940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Aug 04 18:23:17.459259 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.459008    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-jczgf\""
Aug 04 18:23:17.459259 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.459042    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Aug 04 18:23:17.459259 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.459063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Aug 04 18:23:17.464418 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.464393    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"]
Aug 04 18:23:17.467927 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.467901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.468217 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.468196    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"]
Aug 04 18:23:17.483194 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.483162    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"]
Aug 04 18:23:17.519680 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.519680 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.519952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.519952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.519952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.519952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48d4\" (UniqueName: \"kubernetes.io/projected/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-kube-api-access-c48d4\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.519952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xt8\" (UniqueName: \"kubernetes.io/projected/0af9d365-7d9d-484f-be3f-7d4c32916394-kube-api-access-l5xt8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.519952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.519952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.519952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9d365-7d9d-484f-be3f-7d4c32916394-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.519952 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.519945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.520285 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.520041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.621236 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.621236 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.621483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.621483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.621483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.621483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.621483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48d4\" (UniqueName: \"kubernetes.io/projected/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-kube-api-access-c48d4\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.621483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xt8\" (UniqueName: \"kubernetes.io/projected/0af9d365-7d9d-484f-be3f-7d4c32916394-kube-api-access-l5xt8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.621483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.621483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.621483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9d365-7d9d-484f-be3f-7d4c32916394-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.621955 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.621955 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.621955 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.621955 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.621807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.622172 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.622041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.622172 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.622147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.622172 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.622156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.623745 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.623719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.623877 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.623808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.624113 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.624096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9d365-7d9d-484f-be3f-7d4c32916394-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.624154 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.624136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.630201 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.630179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xt8\" (UniqueName: \"kubernetes.io/projected/0af9d365-7d9d-484f-be3f-7d4c32916394-kube-api-access-l5xt8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.630330 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.630307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48d4\" (UniqueName: \"kubernetes.io/projected/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-kube-api-access-c48d4\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.766622 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.766497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:17.779976 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.779890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:17.908720 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.908685    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"]
Aug 04 18:23:17.912012 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:23:17.911985    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebeb3c3_0045_4202_8f02_bcef5dc1b943.slice/crio-77ebd5b78e116da34e8fd2f9a8410c748bb4631aa73d8b660bd3daa3b109779d WatchSource:0}: Error finding container 77ebd5b78e116da34e8fd2f9a8410c748bb4631aa73d8b660bd3daa3b109779d: Status 404 returned error can't find the container with id 77ebd5b78e116da34e8fd2f9a8410c748bb4631aa73d8b660bd3daa3b109779d
Aug 04 18:23:17.914011 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.913992    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:23:17.928159 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:17.928140    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"]
Aug 04 18:23:17.929950 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:23:17.929929    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af9d365_7d9d_484f_be3f_7d4c32916394.slice/crio-36e5dd13081c137ea992af5bbe34d937f87dc9862a18fd568762beba7af942f1 WatchSource:0}: Error finding container 36e5dd13081c137ea992af5bbe34d937f87dc9862a18fd568762beba7af942f1: Status 404 returned error can't find the container with id 36e5dd13081c137ea992af5bbe34d937f87dc9862a18fd568762beba7af942f1
Aug 04 18:23:18.908504 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:18.908409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj" event={"ID":"0af9d365-7d9d-484f-be3f-7d4c32916394","Type":"ContainerStarted","Data":"cddf3e0ff6ee6d80a1032d88135d8f7a2f11ce67dc14d278d08f0bc88bba46c8"}
Aug 04 18:23:18.908504 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:18.908471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj" event={"ID":"0af9d365-7d9d-484f-be3f-7d4c32916394","Type":"ContainerStarted","Data":"36e5dd13081c137ea992af5bbe34d937f87dc9862a18fd568762beba7af942f1"}
Aug 04 18:23:18.909771 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:18.909747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" event={"ID":"3ebeb3c3-0045-4202-8f02-bcef5dc1b943","Type":"ContainerStarted","Data":"77ebd5b78e116da34e8fd2f9a8410c748bb4631aa73d8b660bd3daa3b109779d"}
Aug 04 18:23:18.928569 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:18.928509    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj" podStartSLOduration=1.92848909 podStartE2EDuration="1.92848909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:23:18.926709583 +0000 UTC m=+1406.860779483" watchObservedRunningTime="2026-08-04 18:23:18.92848909 +0000 UTC m=+1406.862558991"
Aug 04 18:23:19.780646 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:19.780589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:20.922764 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:20.922731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" event={"ID":"3ebeb3c3-0045-4202-8f02-bcef5dc1b943","Type":"ContainerStarted","Data":"3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa"}
Aug 04 18:23:20.923206 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:20.922818    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:21.928139 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:21.928107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" event={"ID":"3ebeb3c3-0045-4202-8f02-bcef5dc1b943","Type":"ContainerStarted","Data":"033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c"}
Aug 04 18:23:21.948047 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:21.948000    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" podStartSLOduration=2.73079745 podStartE2EDuration="4.947983398s" totalImagesPullingTime="2.217185948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:23:21.946342107 +0000 UTC m=+1409.880412033" watchObservedRunningTime="2026-08-04 18:23:21.947983398 +0000 UTC m=+1409.882053296"
Aug 04 18:23:22.767267 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:22.767223    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:27.767291 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:27.767255    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:27.779420 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:27.779396    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:27.780549 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:27.780529    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:27.792420 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:27.792398    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:27.962173 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:27.962142    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:27.962958 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:27.962929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:38.970685 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:38.970654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:51.746647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:51.746536    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"]
Aug 04 18:23:51.754947 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:51.754920    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"]
Aug 04 18:23:51.758592 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:23:51.758558    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:23:51.758592 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:23:51.758592 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:23:51.758592 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c" cmd=["/bin/sleep","15"]
Aug 04 18:23:51.758893 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:51.758634    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" message="" err=<
Aug 04 18:23:51.758893 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:23:51.758893 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:23:51.758893 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:23:51.758893 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:23:51.758652    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:23:51.758893 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:23:51.758893 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:23:51.758893 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="main" containerID="cri-o://033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c"
Aug 04 18:23:51.758893 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:51.758693    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="main" containerID="cri-o://033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c" gracePeriod=60
Aug 04 18:23:51.767923 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:23:51.767885    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:23:51.767923 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:23:51.767923 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:23:51.767923 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="cddf3e0ff6ee6d80a1032d88135d8f7a2f11ce67dc14d278d08f0bc88bba46c8" cmd=["/bin/sleep","15"]
Aug 04 18:23:51.768197 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:51.767943    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj" message="" err=<
Aug 04 18:23:51.768197 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:23:51.768197 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:23:51.768197 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:23:51.768197 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:23:51.767959    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:23:51.768197 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:23:51.768197 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:23:51.768197 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj" podUID="0af9d365-7d9d-484f-be3f-7d4c32916394" containerName="main" containerID="cri-o://cddf3e0ff6ee6d80a1032d88135d8f7a2f11ce67dc14d278d08f0bc88bba46c8"
Aug 04 18:23:51.768197 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:51.768010    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj" podUID="0af9d365-7d9d-484f-be3f-7d4c32916394" containerName="main" containerID="cri-o://cddf3e0ff6ee6d80a1032d88135d8f7a2f11ce67dc14d278d08f0bc88bba46c8" gracePeriod=60
Aug 04 18:23:51.952187 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:51.952148    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8001/health\": dial tcp 10.133.0.28:8001: connect: connection refused"
Aug 04 18:23:51.952343 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:51.952156    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj" podUID="0af9d365-7d9d-484f-be3f-7d4c32916394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Aug 04 18:23:51.973764 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:51.973733    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="llm-d-routing-sidecar" containerID="cri-o://3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa" gracePeriod=60
Aug 04 18:23:52.045399 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.045312    2581 generic.go:356] "Generic (PLEG): container finished" podID="0af9d365-7d9d-484f-be3f-7d4c32916394" containerID="cddf3e0ff6ee6d80a1032d88135d8f7a2f11ce67dc14d278d08f0bc88bba46c8" exitCode=0
Aug 04 18:23:52.045399 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.045370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj" event={"ID":"0af9d365-7d9d-484f-be3f-7d4c32916394","Type":"ContainerDied","Data":"cddf3e0ff6ee6d80a1032d88135d8f7a2f11ce67dc14d278d08f0bc88bba46c8"}
Aug 04 18:23:52.047518 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.047495    2581 generic.go:356] "Generic (PLEG): container finished" podID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerID="033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c" exitCode=0
Aug 04 18:23:52.047669 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.047551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" event={"ID":"3ebeb3c3-0045-4202-8f02-bcef5dc1b943","Type":"ContainerDied","Data":"033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c"}
Aug 04 18:23:52.111695 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.111663    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:52.210354 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.210329    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9d365-7d9d-484f-be3f-7d4c32916394-tls-certs\") pod \"0af9d365-7d9d-484f-be3f-7d4c32916394\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") "
Aug 04 18:23:52.210494 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.210366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-tmp-dir\") pod \"0af9d365-7d9d-484f-be3f-7d4c32916394\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") "
Aug 04 18:23:52.210494 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.210400    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-model-cache\") pod \"0af9d365-7d9d-484f-be3f-7d4c32916394\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") "
Aug 04 18:23:52.210494 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.210424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5xt8\" (UniqueName: \"kubernetes.io/projected/0af9d365-7d9d-484f-be3f-7d4c32916394-kube-api-access-l5xt8\") pod \"0af9d365-7d9d-484f-be3f-7d4c32916394\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") "
Aug 04 18:23:52.210494 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.210459    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-home\") pod \"0af9d365-7d9d-484f-be3f-7d4c32916394\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") "
Aug 04 18:23:52.210741 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.210498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-dshm\") pod \"0af9d365-7d9d-484f-be3f-7d4c32916394\" (UID: \"0af9d365-7d9d-484f-be3f-7d4c32916394\") "
Aug 04 18:23:52.210741 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.210676    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0af9d365-7d9d-484f-be3f-7d4c32916394" (UID: "0af9d365-7d9d-484f-be3f-7d4c32916394"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:52.210741 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.210687    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-model-cache" (OuterVolumeSpecName: "model-cache") pod "0af9d365-7d9d-484f-be3f-7d4c32916394" (UID: "0af9d365-7d9d-484f-be3f-7d4c32916394"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:52.210884 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.210781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-home" (OuterVolumeSpecName: "home") pod "0af9d365-7d9d-484f-be3f-7d4c32916394" (UID: "0af9d365-7d9d-484f-be3f-7d4c32916394"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:52.212569 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.212545    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-dshm" (OuterVolumeSpecName: "dshm") pod "0af9d365-7d9d-484f-be3f-7d4c32916394" (UID: "0af9d365-7d9d-484f-be3f-7d4c32916394"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:52.212711 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.212571    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af9d365-7d9d-484f-be3f-7d4c32916394-kube-api-access-l5xt8" (OuterVolumeSpecName: "kube-api-access-l5xt8") pod "0af9d365-7d9d-484f-be3f-7d4c32916394" (UID: "0af9d365-7d9d-484f-be3f-7d4c32916394"). InnerVolumeSpecName "kube-api-access-l5xt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:23:52.212711 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.212585    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af9d365-7d9d-484f-be3f-7d4c32916394-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0af9d365-7d9d-484f-be3f-7d4c32916394" (UID: "0af9d365-7d9d-484f-be3f-7d4c32916394"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:23:52.216428 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.216408    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:52.311322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311224    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c48d4\" (UniqueName: \"kubernetes.io/projected/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-kube-api-access-c48d4\") pod \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") "
Aug 04 18:23:52.311322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311290    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tmp-dir\") pod \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") "
Aug 04 18:23:52.311322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311318    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-home\") pod \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") "
Aug 04 18:23:52.311644 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311337    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-model-cache\") pod \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") "
Aug 04 18:23:52.311644 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-dshm\") pod \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") "
Aug 04 18:23:52.311644 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311398    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tls-certs\") pod \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\" (UID: \"3ebeb3c3-0045-4202-8f02-bcef5dc1b943\") "
Aug 04 18:23:52.311644 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311595    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xt8\" (UniqueName: \"kubernetes.io/projected/0af9d365-7d9d-484f-be3f-7d4c32916394-kube-api-access-l5xt8\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.311644 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311639    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.311836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311654    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.311836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311668    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9d365-7d9d-484f-be3f-7d4c32916394-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.311836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311665    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ebeb3c3-0045-4202-8f02-bcef5dc1b943" (UID: "3ebeb3c3-0045-4202-8f02-bcef5dc1b943"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:52.311836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311677    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ebeb3c3-0045-4202-8f02-bcef5dc1b943" (UID: "3ebeb3c3-0045-4202-8f02-bcef5dc1b943"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:52.311836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311694    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-home" (OuterVolumeSpecName: "home") pod "3ebeb3c3-0045-4202-8f02-bcef5dc1b943" (UID: "3ebeb3c3-0045-4202-8f02-bcef5dc1b943"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:52.311836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311710    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.311836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.311737    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af9d365-7d9d-484f-be3f-7d4c32916394-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.313417 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.313391    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-kube-api-access-c48d4" (OuterVolumeSpecName: "kube-api-access-c48d4") pod "3ebeb3c3-0045-4202-8f02-bcef5dc1b943" (UID: "3ebeb3c3-0045-4202-8f02-bcef5dc1b943"). InnerVolumeSpecName "kube-api-access-c48d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:23:52.313549 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.313527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-dshm" (OuterVolumeSpecName: "dshm") pod "3ebeb3c3-0045-4202-8f02-bcef5dc1b943" (UID: "3ebeb3c3-0045-4202-8f02-bcef5dc1b943"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:52.313889 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.313867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ebeb3c3-0045-4202-8f02-bcef5dc1b943" (UID: "3ebeb3c3-0045-4202-8f02-bcef5dc1b943"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:23:52.412588 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.412542    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c48d4\" (UniqueName: \"kubernetes.io/projected/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-kube-api-access-c48d4\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.412588 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.412583    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.412588 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.412596    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.412588 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.412627    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.412588 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.412635    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:52.412920 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:52.412643    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebeb3c3-0045-4202-8f02-bcef5dc1b943-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:23:53.052758 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.052727    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"
Aug 04 18:23:53.053227 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.052724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj" event={"ID":"0af9d365-7d9d-484f-be3f-7d4c32916394","Type":"ContainerDied","Data":"36e5dd13081c137ea992af5bbe34d937f87dc9862a18fd568762beba7af942f1"}
Aug 04 18:23:53.053227 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.052864    2581 scope.go:117] "RemoveContainer" containerID="cddf3e0ff6ee6d80a1032d88135d8f7a2f11ce67dc14d278d08f0bc88bba46c8"
Aug 04 18:23:53.054391 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.054365    2581 generic.go:356] "Generic (PLEG): container finished" podID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerID="3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa" exitCode=0
Aug 04 18:23:53.054502 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.054428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" event={"ID":"3ebeb3c3-0045-4202-8f02-bcef5dc1b943","Type":"ContainerDied","Data":"3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa"}
Aug 04 18:23:53.054502 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.054480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g" event={"ID":"3ebeb3c3-0045-4202-8f02-bcef5dc1b943","Type":"ContainerDied","Data":"77ebd5b78e116da34e8fd2f9a8410c748bb4631aa73d8b660bd3daa3b109779d"}
Aug 04 18:23:53.054502 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.054433    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"
Aug 04 18:23:53.061727 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.061708    2581 scope.go:117] "RemoveContainer" containerID="033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c"
Aug 04 18:23:53.068630 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.068613    2581 scope.go:117] "RemoveContainer" containerID="3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa"
Aug 04 18:23:53.075109 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.075089    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"]
Aug 04 18:23:53.075391 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.075374    2581 scope.go:117] "RemoveContainer" containerID="033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c"
Aug 04 18:23:53.075648 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:23:53.075632    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c\": container with ID starting with 033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c not found: ID does not exist" containerID="033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c"
Aug 04 18:23:53.075707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.075657    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c"} err="failed to get container status \"033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c\": rpc error: code = NotFound desc = could not find container \"033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c\": container with ID starting with 033b74efb3ae491acf1ff2cd0e467e40f96e7fbea096222eeaf401d78d79614c not found: ID does not exist"
Aug 04 18:23:53.075707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.075675    2581 scope.go:117] "RemoveContainer" containerID="3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa"
Aug 04 18:23:53.075893 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:23:53.075876    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa\": container with ID starting with 3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa not found: ID does not exist" containerID="3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa"
Aug 04 18:23:53.075935 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.075899    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa"} err="failed to get container status \"3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa\": rpc error: code = NotFound desc = could not find container \"3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa\": container with ID starting with 3cd30501756627d32aa2ef9c881e26004904d3914862f1111af44ae7b12dd8aa not found: ID does not exist"
Aug 04 18:23:53.081102 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.081081    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hjd9g"]
Aug 04 18:23:53.095341 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.095322    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"]
Aug 04 18:23:53.105173 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:53.105155    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fss5bj"]
Aug 04 18:23:54.546040 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:54.546003    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af9d365-7d9d-484f-be3f-7d4c32916394" path="/var/lib/kubelet/pods/0af9d365-7d9d-484f-be3f-7d4c32916394/volumes"
Aug 04 18:23:54.546409 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:23:54.546366    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" path="/var/lib/kubelet/pods/3ebeb3c3-0045-4202-8f02-bcef5dc1b943/volumes"
Aug 04 18:25:14.792921 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.792838    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"]
Aug 04 18:25:14.793374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.793101    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="main"
Aug 04 18:25:14.793374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.793112    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="main"
Aug 04 18:25:14.793374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.793127    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="llm-d-routing-sidecar"
Aug 04 18:25:14.793374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.793133    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="llm-d-routing-sidecar"
Aug 04 18:25:14.793374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.793145    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af9d365-7d9d-484f-be3f-7d4c32916394" containerName="main"
Aug 04 18:25:14.793374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.793151    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af9d365-7d9d-484f-be3f-7d4c32916394" containerName="main"
Aug 04 18:25:14.793374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.793192    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af9d365-7d9d-484f-be3f-7d4c32916394" containerName="main"
Aug 04 18:25:14.793374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.793199    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="llm-d-routing-sidecar"
Aug 04 18:25:14.793374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.793204    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebeb3c3-0045-4202-8f02-bcef5dc1b943" containerName="main"
Aug 04 18:25:14.795133 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.795116    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.800382 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.800346    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Aug 04 18:25:14.801544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.801529    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Aug 04 18:25:14.801814 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.801793    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:25:14.801814 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.801793    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Aug 04 18:25:14.801978 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.801833    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-cjq6s\""
Aug 04 18:25:14.802258 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.802238    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Aug 04 18:25:14.835658 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.835632    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"]
Aug 04 18:25:14.843338 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.843311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.843481 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.843348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-home\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.843481 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.843375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvn6\" (UniqueName: \"kubernetes.io/projected/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kube-api-access-fjvn6\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.843481 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.843450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.843693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.843487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.843693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.843508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.843693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.843528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-model-cache\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.843693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.843548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-dshm\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944359 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944572 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944572 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944572 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-model-cache\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944572 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-dshm\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944572 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944572 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-home\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944963 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvn6\" (UniqueName: \"kubernetes.io/projected/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kube-api-access-fjvn6\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944963 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.944963 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.944890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-model-cache\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.945205 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.945185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.945264 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.945243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-home\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.945303 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.945247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.946902 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.946882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-dshm\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.947571 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.947539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:14.963646 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:14.963597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvn6\" (UniqueName: \"kubernetes.io/projected/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kube-api-access-fjvn6\") pod \"tokenizer-clean-path-test-kserve-56649fd646-hnvlx\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:15.104846 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:15.104738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:15.261520 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:15.261492    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"]
Aug 04 18:25:15.263429 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:25:15.263397    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f88bdc_ac7c_4194_b508_34b8d865f4b2.slice/crio-aba59eae5329f3bf72d351cb4484e9b7fc592cbf590e398dfa4b392508180096 WatchSource:0}: Error finding container aba59eae5329f3bf72d351cb4484e9b7fc592cbf590e398dfa4b392508180096: Status 404 returned error can't find the container with id aba59eae5329f3bf72d351cb4484e9b7fc592cbf590e398dfa4b392508180096
Aug 04 18:25:15.359443 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:15.359357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" event={"ID":"e7f88bdc-ac7c-4194-b508-34b8d865f4b2","Type":"ContainerStarted","Data":"28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9"}
Aug 04 18:25:15.359443 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:15.359409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" event={"ID":"e7f88bdc-ac7c-4194-b508-34b8d865f4b2","Type":"ContainerStarted","Data":"aba59eae5329f3bf72d351cb4484e9b7fc592cbf590e398dfa4b392508180096"}
Aug 04 18:25:18.372015 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:18.371980    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" containerID="28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9" exitCode=0
Aug 04 18:25:18.372393 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:18.372053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" event={"ID":"e7f88bdc-ac7c-4194-b508-34b8d865f4b2","Type":"ContainerDied","Data":"28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9"}
Aug 04 18:25:19.377186 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:19.377151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" event={"ID":"e7f88bdc-ac7c-4194-b508-34b8d865f4b2","Type":"ContainerStarted","Data":"cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce"}
Aug 04 18:25:19.403463 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:19.403411    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" podStartSLOduration=3.347450862 podStartE2EDuration="5.40339716s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.055946298s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:25:19.401554611 +0000 UTC m=+1527.335624510" watchObservedRunningTime="2026-08-04 18:25:19.40339716 +0000 UTC m=+1527.337467058"
Aug 04 18:25:20.105107 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:20.105073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:25.105281 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:25.105243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:25.117905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:25.117881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:25:25.408916 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:25:25.408879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:26:30.973317 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:30.973287    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"]
Aug 04 18:26:30.984443 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:26:30.984401    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:26:30.984443 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:26:30.984443 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:26:30.984443 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce" cmd=["/bin/sleep","15"]
Aug 04 18:26:30.984717 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:30.984464    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" message="" err=<
Aug 04 18:26:30.984717 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:26:30.984717 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:26:30.984717 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:26:30.984717 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:26:30.984481    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:26:30.984717 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:26:30.984717 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:26:30.984717 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" podUID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" containerName="main" containerID="cri-o://cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce"
Aug 04 18:26:30.984717 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:30.984537    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" podUID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" containerName="main" containerID="cri-o://cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce" gracePeriod=60
Aug 04 18:26:31.345766 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.345741    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:26:31.474173 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474143    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-home\") pod \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") "
Aug 04 18:26:31.474337 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474210    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tmp-dir\") pod \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") "
Aug 04 18:26:31.474337 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-model-cache\") pod \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") "
Aug 04 18:26:31.474337 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474264    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tls-certs\") pod \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") "
Aug 04 18:26:31.474337 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474290    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kserve-provision-location\") pod \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") "
Aug 04 18:26:31.474337 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474316    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-cabundle-cert\") pod \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") "
Aug 04 18:26:31.474565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474357    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-dshm\") pod \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") "
Aug 04 18:26:31.474565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjvn6\" (UniqueName: \"kubernetes.io/projected/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kube-api-access-fjvn6\") pod \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\" (UID: \"e7f88bdc-ac7c-4194-b508-34b8d865f4b2\") "
Aug 04 18:26:31.474565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474491    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-home" (OuterVolumeSpecName: "home") pod "e7f88bdc-ac7c-4194-b508-34b8d865f4b2" (UID: "e7f88bdc-ac7c-4194-b508-34b8d865f4b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.474565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474507    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7f88bdc-ac7c-4194-b508-34b8d865f4b2" (UID: "e7f88bdc-ac7c-4194-b508-34b8d865f4b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.474565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474547    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7f88bdc-ac7c-4194-b508-34b8d865f4b2" (UID: "e7f88bdc-ac7c-4194-b508-34b8d865f4b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.474840 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474670    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.474840 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474687    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.474840 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.474702    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.475086 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.475063    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7f88bdc-ac7c-4194-b508-34b8d865f4b2" (UID: "e7f88bdc-ac7c-4194-b508-34b8d865f4b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:26:31.476532 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.476511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7f88bdc-ac7c-4194-b508-34b8d865f4b2" (UID: "e7f88bdc-ac7c-4194-b508-34b8d865f4b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:26:31.476648 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.476582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kube-api-access-fjvn6" (OuterVolumeSpecName: "kube-api-access-fjvn6") pod "e7f88bdc-ac7c-4194-b508-34b8d865f4b2" (UID: "e7f88bdc-ac7c-4194-b508-34b8d865f4b2"). InnerVolumeSpecName "kube-api-access-fjvn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:26:31.476771 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.476749    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-dshm" (OuterVolumeSpecName: "dshm") pod "e7f88bdc-ac7c-4194-b508-34b8d865f4b2" (UID: "e7f88bdc-ac7c-4194-b508-34b8d865f4b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.494558 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.494529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7f88bdc-ac7c-4194-b508-34b8d865f4b2" (UID: "e7f88bdc-ac7c-4194-b508-34b8d865f4b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.575088 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.575010    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.575088 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.575036    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.575088 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.575047    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.575088 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.575056    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.575088 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.575064    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjvn6\" (UniqueName: \"kubernetes.io/projected/e7f88bdc-ac7c-4194-b508-34b8d865f4b2-kube-api-access-fjvn6\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.634889 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.634858    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" containerID="cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce" exitCode=0
Aug 04 18:26:31.635031 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.634913    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"
Aug 04 18:26:31.635031 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.634916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" event={"ID":"e7f88bdc-ac7c-4194-b508-34b8d865f4b2","Type":"ContainerDied","Data":"cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce"}
Aug 04 18:26:31.635121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.635039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx" event={"ID":"e7f88bdc-ac7c-4194-b508-34b8d865f4b2","Type":"ContainerDied","Data":"aba59eae5329f3bf72d351cb4484e9b7fc592cbf590e398dfa4b392508180096"}
Aug 04 18:26:31.635121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.635063    2581 scope.go:117] "RemoveContainer" containerID="cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce"
Aug 04 18:26:31.644394 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.644376    2581 scope.go:117] "RemoveContainer" containerID="28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9"
Aug 04 18:26:31.651435 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.651419    2581 scope.go:117] "RemoveContainer" containerID="cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce"
Aug 04 18:26:31.651715 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:26:31.651697    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce\": container with ID starting with cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce not found: ID does not exist" containerID="cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce"
Aug 04 18:26:31.651774 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.651725    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce"} err="failed to get container status \"cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce\": rpc error: code = NotFound desc = could not find container \"cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce\": container with ID starting with cb4df1bfaff1a11778b93c865e0c27c1420d64d4024f0d131c1796ca428b80ce not found: ID does not exist"
Aug 04 18:26:31.651774 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.651741    2581 scope.go:117] "RemoveContainer" containerID="28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9"
Aug 04 18:26:31.651982 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:26:31.651964    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9\": container with ID starting with 28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9 not found: ID does not exist" containerID="28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9"
Aug 04 18:26:31.652026 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.651988    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9"} err="failed to get container status \"28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9\": rpc error: code = NotFound desc = could not find container \"28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9\": container with ID starting with 28d128db53bfb59293fd351e734fd6c10861e81422a32c6d8591cdc03351a2b9 not found: ID does not exist"
Aug 04 18:26:31.658169 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.658145    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"]
Aug 04 18:26:31.664223 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:31.664203    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-hnvlx"]
Aug 04 18:26:32.545294 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:26:32.545259    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" path="/var/lib/kubelet/pods/e7f88bdc-ac7c-4194-b508-34b8d865f4b2/volumes"
Aug 04 18:27:00.839598 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.839565    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"]
Aug 04 18:27:00.839999 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.839853    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" containerName="main"
Aug 04 18:27:00.839999 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.839864    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" containerName="main"
Aug 04 18:27:00.839999 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.839873    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" containerName="storage-initializer"
Aug 04 18:27:00.839999 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.839880    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" containerName="storage-initializer"
Aug 04 18:27:00.839999 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.839931    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f88bdc-ac7c-4194-b508-34b8d865f4b2" containerName="main"
Aug 04 18:27:00.842653 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.842633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:00.845535 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.845511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Aug 04 18:27:00.845671 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.845654    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:27:00.845841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.845815    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Aug 04 18:27:00.846822 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.846806    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Aug 04 18:27:00.846912 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.846832    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-vr6g7\""
Aug 04 18:27:00.847129 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.847110    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Aug 04 18:27:00.852879 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:00.852827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"]
Aug 04 18:27:01.004119 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.004086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.004325 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.004145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-home\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.004325 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.004201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8ee50c-767e-4da9-af42-bc2de2fc563c-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.004325 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.004218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.004325 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.004271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.004325 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.004306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.004325 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.004327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-dshm\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.004504 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.004370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g9vs\" (UniqueName: \"kubernetes.io/projected/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kube-api-access-7g9vs\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105016 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.104931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-home\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105016 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.104978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8ee50c-767e-4da9-af42-bc2de2fc563c-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105227 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105227 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105336 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105336 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-dshm\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105489 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g9vs\" (UniqueName: \"kubernetes.io/projected/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kube-api-access-7g9vs\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105489 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105489 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-home\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105489 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105708 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105708 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.105806 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.105716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8ee50c-767e-4da9-af42-bc2de2fc563c-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.107465 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.107443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-dshm\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.107734 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.107718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.114309 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.114289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g9vs\" (UniqueName: \"kubernetes.io/projected/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kube-api-access-7g9vs\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-zgcrw\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.153244 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.153211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:01.283019 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.282987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"]
Aug 04 18:27:01.285896 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:27:01.285870    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8ee50c_767e_4da9_af42_bc2de2fc563c.slice/crio-3b3d087fca091d2403ced1ce9f077f602d1bba8c08fb18242f499d3f4284d9d8 WatchSource:0}: Error finding container 3b3d087fca091d2403ced1ce9f077f602d1bba8c08fb18242f499d3f4284d9d8: Status 404 returned error can't find the container with id 3b3d087fca091d2403ced1ce9f077f602d1bba8c08fb18242f499d3f4284d9d8
Aug 04 18:27:01.741588 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.741542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" event={"ID":"4d8ee50c-767e-4da9-af42-bc2de2fc563c","Type":"ContainerStarted","Data":"4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b"}
Aug 04 18:27:01.741781 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:01.741676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" event={"ID":"4d8ee50c-767e-4da9-af42-bc2de2fc563c","Type":"ContainerStarted","Data":"3b3d087fca091d2403ced1ce9f077f602d1bba8c08fb18242f499d3f4284d9d8"}
Aug 04 18:27:03.752017 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:03.751934    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerID="4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b" exitCode=0
Aug 04 18:27:03.752390 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:03.752013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" event={"ID":"4d8ee50c-767e-4da9-af42-bc2de2fc563c","Type":"ContainerDied","Data":"4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b"}
Aug 04 18:27:04.757149 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:04.757117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" event={"ID":"4d8ee50c-767e-4da9-af42-bc2de2fc563c","Type":"ContainerStarted","Data":"4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8"}
Aug 04 18:27:04.788524 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:04.788466    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" podStartSLOduration=2.732745941 podStartE2EDuration="4.788450354s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.055704413s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:04.786231082 +0000 UTC m=+1632.720300982" watchObservedRunningTime="2026-08-04 18:27:04.788450354 +0000 UTC m=+1632.722520254"
Aug 04 18:27:05.154043 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:05.154014    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:11.153938 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:11.153887    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:11.166643 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:11.166617    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:11.798543 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:11.798509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:18.796210 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.796179    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"]
Aug 04 18:27:18.798757 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.798741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.801460 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.801430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Aug 04 18:27:18.801558 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.801430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-b96fh\""
Aug 04 18:27:18.802744 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.802724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Aug 04 18:27:18.802840 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.802750    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Aug 04 18:27:18.810777 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.810755    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"]
Aug 04 18:27:18.827854 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.827829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a288747-85f1-4eb3-9fbb-e942215723d4-tls-certs\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.827972 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.827863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-dshm\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.827972 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.827884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pzx\" (UniqueName: \"kubernetes.io/projected/4a288747-85f1-4eb3-9fbb-e942215723d4-kube-api-access-v2pzx\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.827972 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.827901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-home\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.828112 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.827977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-model-cache\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.828112 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.828004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.928908 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.928873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a288747-85f1-4eb3-9fbb-e942215723d4-tls-certs\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.929098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.928922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-dshm\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.929098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.928950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pzx\" (UniqueName: \"kubernetes.io/projected/4a288747-85f1-4eb3-9fbb-e942215723d4-kube-api-access-v2pzx\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.929098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.928973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-home\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.929098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.929016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-model-cache\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.929098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.929044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.929422 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.929392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-home\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.929546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.929448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-model-cache\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.929546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.929506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.931178 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.931159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-dshm\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.931346 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.931329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a288747-85f1-4eb3-9fbb-e942215723d4-tls-certs\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:18.938696 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:18.938676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pzx\" (UniqueName: \"kubernetes.io/projected/4a288747-85f1-4eb3-9fbb-e942215723d4-kube-api-access-v2pzx\") pod \"test-div-v1-kserve-fb566d667-8xpg7\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:19.108910 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:19.108823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:19.241112 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:19.241087    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"]
Aug 04 18:27:19.242100 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:27:19.242077    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a288747_85f1_4eb3_9fbb_e942215723d4.slice/crio-88711bd17e8abc09cfaf4a6f9b3b695de6f0fa4ea58b081655f276cef0bd12ca WatchSource:0}: Error finding container 88711bd17e8abc09cfaf4a6f9b3b695de6f0fa4ea58b081655f276cef0bd12ca: Status 404 returned error can't find the container with id 88711bd17e8abc09cfaf4a6f9b3b695de6f0fa4ea58b081655f276cef0bd12ca
Aug 04 18:27:19.816843 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:19.816786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7" event={"ID":"4a288747-85f1-4eb3-9fbb-e942215723d4","Type":"ContainerStarted","Data":"88711bd17e8abc09cfaf4a6f9b3b695de6f0fa4ea58b081655f276cef0bd12ca"}
Aug 04 18:27:20.822460 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:20.822379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7" event={"ID":"4a288747-85f1-4eb3-9fbb-e942215723d4","Type":"ContainerStarted","Data":"138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217"}
Aug 04 18:27:20.840036 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:20.839992    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7" podStartSLOduration=1.68621631 podStartE2EDuration="2.839976006s" totalImagesPullingTime="1.153759696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:20.839481355 +0000 UTC m=+1648.773551254" watchObservedRunningTime="2026-08-04 18:27:20.839976006 +0000 UTC m=+1648.774045904"
Aug 04 18:27:21.109769 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:21.109676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:29.109027 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:29.108988    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:29.121574 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:29.121541    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:29.866560 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:29.866527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:27:56.565938 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.565900    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"]
Aug 04 18:27:56.575354 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:27:56.575311    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:27:56.575354 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:27:56.575354 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:27:56.575354 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8" cmd=["/bin/sleep","15"]
Aug 04 18:27:56.575835 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.575376    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" message="" err=<
Aug 04 18:27:56.575835 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:27:56.575835 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:27:56.575835 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:27:56.575835 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:27:56.575397    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:27:56.575835 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:27:56.575835 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:27:56.575835 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" podUID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerName="main" containerID="cri-o://4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8"
Aug 04 18:27:56.575835 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.575443    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" podUID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerName="main" containerID="cri-o://4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8" gracePeriod=60
Aug 04 18:27:56.782660 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.782620    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" podUID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Aug 04 18:27:56.919472 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.919446    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:56.955799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.955766    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerID="4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8" exitCode=0
Aug 04 18:27:56.955959 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.955826    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"
Aug 04 18:27:56.955959 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.955865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" event={"ID":"4d8ee50c-767e-4da9-af42-bc2de2fc563c","Type":"ContainerDied","Data":"4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8"}
Aug 04 18:27:56.955959 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.955934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw" event={"ID":"4d8ee50c-767e-4da9-af42-bc2de2fc563c","Type":"ContainerDied","Data":"3b3d087fca091d2403ced1ce9f077f602d1bba8c08fb18242f499d3f4284d9d8"}
Aug 04 18:27:56.956078 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.955967    2581 scope.go:117] "RemoveContainer" containerID="4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8"
Aug 04 18:27:56.965303 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.965276    2581 scope.go:117] "RemoveContainer" containerID="4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b"
Aug 04 18:27:56.974059 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.974036    2581 scope.go:117] "RemoveContainer" containerID="4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8"
Aug 04 18:27:56.974314 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:27:56.974295    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8\": container with ID starting with 4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8 not found: ID does not exist" containerID="4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8"
Aug 04 18:27:56.974375 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.974326    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8"} err="failed to get container status \"4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8\": rpc error: code = NotFound desc = could not find container \"4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8\": container with ID starting with 4e14460434ebda26730c2ec074febcb48151704f05b2ca7c08e30c063ad324e8 not found: ID does not exist"
Aug 04 18:27:56.974375 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.974346    2581 scope.go:117] "RemoveContainer" containerID="4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b"
Aug 04 18:27:56.974628 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:27:56.974580    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b\": container with ID starting with 4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b not found: ID does not exist" containerID="4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b"
Aug 04 18:27:56.974736 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:56.974683    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b"} err="failed to get container status \"4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b\": rpc error: code = NotFound desc = could not find container \"4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b\": container with ID starting with 4716b36d1d0bf4fc17e1069561f61979fca4ce588aac2441963cf504dc8d6f1b not found: ID does not exist"
Aug 04 18:27:57.058279 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.058247    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kserve-provision-location\") pod \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") "
Aug 04 18:27:57.058279 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.058286    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-dshm\") pod \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") "
Aug 04 18:27:57.058531 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.058316    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-home\") pod \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") "
Aug 04 18:27:57.058531 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.058372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tmp-dir\") pod \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") "
Aug 04 18:27:57.058531 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.058396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tls-certs\") pod \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") "
Aug 04 18:27:57.058531 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.058445    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-model-cache\") pod \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") "
Aug 04 18:27:57.058531 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.058487    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g9vs\" (UniqueName: \"kubernetes.io/projected/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kube-api-access-7g9vs\") pod \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") "
Aug 04 18:27:57.058531 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.058516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8ee50c-767e-4da9-af42-bc2de2fc563c-cabundle-cert\") pod \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\" (UID: \"4d8ee50c-767e-4da9-af42-bc2de2fc563c\") "
Aug 04 18:27:57.058883 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.058720    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d8ee50c-767e-4da9-af42-bc2de2fc563c" (UID: "4d8ee50c-767e-4da9-af42-bc2de2fc563c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.059174 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.059043    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8ee50c-767e-4da9-af42-bc2de2fc563c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d8ee50c-767e-4da9-af42-bc2de2fc563c" (UID: "4d8ee50c-767e-4da9-af42-bc2de2fc563c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:27:57.059174 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.059087    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d8ee50c-767e-4da9-af42-bc2de2fc563c" (UID: "4d8ee50c-767e-4da9-af42-bc2de2fc563c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.059431 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.059397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-home" (OuterVolumeSpecName: "home") pod "4d8ee50c-767e-4da9-af42-bc2de2fc563c" (UID: "4d8ee50c-767e-4da9-af42-bc2de2fc563c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.061733 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.061689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d8ee50c-767e-4da9-af42-bc2de2fc563c" (UID: "4d8ee50c-767e-4da9-af42-bc2de2fc563c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:27:57.061733 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.061693    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kube-api-access-7g9vs" (OuterVolumeSpecName: "kube-api-access-7g9vs") pod "4d8ee50c-767e-4da9-af42-bc2de2fc563c" (UID: "4d8ee50c-767e-4da9-af42-bc2de2fc563c"). InnerVolumeSpecName "kube-api-access-7g9vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:27:57.061958 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.061781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-dshm" (OuterVolumeSpecName: "dshm") pod "4d8ee50c-767e-4da9-af42-bc2de2fc563c" (UID: "4d8ee50c-767e-4da9-af42-bc2de2fc563c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.079712 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.079682    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d8ee50c-767e-4da9-af42-bc2de2fc563c" (UID: "4d8ee50c-767e-4da9-af42-bc2de2fc563c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.159496 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.159467    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.159496 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.159492    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g9vs\" (UniqueName: \"kubernetes.io/projected/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kube-api-access-7g9vs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.159717 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.159505    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8ee50c-767e-4da9-af42-bc2de2fc563c-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.159717 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.159514    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.159717 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.159522    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.159717 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.159531    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.159717 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.159559    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.159717 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.159567    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8ee50c-767e-4da9-af42-bc2de2fc563c-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.280996 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.280969    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"]
Aug 04 18:27:57.289789 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:57.289764    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-zgcrw"]
Aug 04 18:27:58.545182 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:58.545148    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" path="/var/lib/kubelet/pods/4d8ee50c-767e-4da9-af42-bc2de2fc563c/volumes"
Aug 04 18:27:58.938567 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:27:58.938537    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"]
Aug 04 18:28:13.956729 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:13.956663    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7" podUID="4a288747-85f1-4eb3-9fbb-e942215723d4" containerName="main" containerID="cri-o://138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217" gracePeriod=45
Aug 04 18:28:14.203027 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.203003    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:28:14.279246 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.279132    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2pzx\" (UniqueName: \"kubernetes.io/projected/4a288747-85f1-4eb3-9fbb-e942215723d4-kube-api-access-v2pzx\") pod \"4a288747-85f1-4eb3-9fbb-e942215723d4\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") "
Aug 04 18:28:14.279246 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.279165    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a288747-85f1-4eb3-9fbb-e942215723d4-tls-certs\") pod \"4a288747-85f1-4eb3-9fbb-e942215723d4\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") "
Aug 04 18:28:14.279246 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.279190    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-dshm\") pod \"4a288747-85f1-4eb3-9fbb-e942215723d4\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") "
Aug 04 18:28:14.279246 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.279207    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-home\") pod \"4a288747-85f1-4eb3-9fbb-e942215723d4\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") "
Aug 04 18:28:14.279246 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.279228    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-tmp-dir\") pod \"4a288747-85f1-4eb3-9fbb-e942215723d4\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") "
Aug 04 18:28:14.279641 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.279297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-model-cache\") pod \"4a288747-85f1-4eb3-9fbb-e942215723d4\" (UID: \"4a288747-85f1-4eb3-9fbb-e942215723d4\") "
Aug 04 18:28:14.279641 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.279505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-home" (OuterVolumeSpecName: "home") pod "4a288747-85f1-4eb3-9fbb-e942215723d4" (UID: "4a288747-85f1-4eb3-9fbb-e942215723d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:28:14.279641 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.279535    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a288747-85f1-4eb3-9fbb-e942215723d4" (UID: "4a288747-85f1-4eb3-9fbb-e942215723d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:28:14.279751 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.279686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a288747-85f1-4eb3-9fbb-e942215723d4" (UID: "4a288747-85f1-4eb3-9fbb-e942215723d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:28:14.281556 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.281524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-dshm" (OuterVolumeSpecName: "dshm") pod "4a288747-85f1-4eb3-9fbb-e942215723d4" (UID: "4a288747-85f1-4eb3-9fbb-e942215723d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:28:14.281679 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.281593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a288747-85f1-4eb3-9fbb-e942215723d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a288747-85f1-4eb3-9fbb-e942215723d4" (UID: "4a288747-85f1-4eb3-9fbb-e942215723d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:28:14.281679 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.281627    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a288747-85f1-4eb3-9fbb-e942215723d4-kube-api-access-v2pzx" (OuterVolumeSpecName: "kube-api-access-v2pzx") pod "4a288747-85f1-4eb3-9fbb-e942215723d4" (UID: "4a288747-85f1-4eb3-9fbb-e942215723d4"). InnerVolumeSpecName "kube-api-access-v2pzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:28:14.380310 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.380276    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2pzx\" (UniqueName: \"kubernetes.io/projected/4a288747-85f1-4eb3-9fbb-e942215723d4-kube-api-access-v2pzx\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.380310 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.380304    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a288747-85f1-4eb3-9fbb-e942215723d4-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.380310 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.380314    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.380532 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.380325    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.380532 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.380332    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.380532 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:14.380340    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a288747-85f1-4eb3-9fbb-e942215723d4-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:28:15.026079 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:15.026043    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a288747-85f1-4eb3-9fbb-e942215723d4" containerID="138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217" exitCode=0
Aug 04 18:28:15.026450 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:15.026087    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"
Aug 04 18:28:15.026450 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:15.026115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7" event={"ID":"4a288747-85f1-4eb3-9fbb-e942215723d4","Type":"ContainerDied","Data":"138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217"}
Aug 04 18:28:15.026450 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:15.026157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7" event={"ID":"4a288747-85f1-4eb3-9fbb-e942215723d4","Type":"ContainerDied","Data":"88711bd17e8abc09cfaf4a6f9b3b695de6f0fa4ea58b081655f276cef0bd12ca"}
Aug 04 18:28:15.026450 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:15.026178    2581 scope.go:117] "RemoveContainer" containerID="138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217"
Aug 04 18:28:15.033984 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:15.033968    2581 scope.go:117] "RemoveContainer" containerID="138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217"
Aug 04 18:28:15.034274 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:28:15.034256    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217\": container with ID starting with 138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217 not found: ID does not exist" containerID="138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217"
Aug 04 18:28:15.034341 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:15.034286    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217"} err="failed to get container status \"138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217\": rpc error: code = NotFound desc = could not find container \"138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217\": container with ID starting with 138107485df9942665cd4dad2b720704ff4d214822321557684a8456de794217 not found: ID does not exist"
Aug 04 18:28:15.045670 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:15.045643    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"]
Aug 04 18:28:15.053215 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:15.053192    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8xpg7"]
Aug 04 18:28:16.544867 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:16.544831    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a288747-85f1-4eb3-9fbb-e942215723d4" path="/var/lib/kubelet/pods/4a288747-85f1-4eb3-9fbb-e942215723d4/volumes"
Aug 04 18:28:42.019534 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.019497    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"]
Aug 04 18:28:42.019923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.019769    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a288747-85f1-4eb3-9fbb-e942215723d4" containerName="main"
Aug 04 18:28:42.019923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.019781    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a288747-85f1-4eb3-9fbb-e942215723d4" containerName="main"
Aug 04 18:28:42.019923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.019792    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerName="main"
Aug 04 18:28:42.019923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.019798    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerName="main"
Aug 04 18:28:42.019923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.019815    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerName="storage-initializer"
Aug 04 18:28:42.019923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.019822    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerName="storage-initializer"
Aug 04 18:28:42.019923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.019868    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a288747-85f1-4eb3-9fbb-e942215723d4" containerName="main"
Aug 04 18:28:42.019923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.019877    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8ee50c-767e-4da9-af42-bc2de2fc563c" containerName="main"
Aug 04 18:28:42.022856 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.022838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.028619 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.028568    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Aug 04 18:28:42.028722 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.028590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Aug 04 18:28:42.028722 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.028596    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:28:42.029689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.029669    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Aug 04 18:28:42.029771 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.029717    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-9qrb5\""
Aug 04 18:28:42.029859 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.029839    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Aug 04 18:28:42.034839 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.034816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"]
Aug 04 18:28:42.083683 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.083649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5gt\" (UniqueName: \"kubernetes.io/projected/c2b6341c-d538-4cf5-a311-1d1735f601f2-kube-api-access-ks5gt\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.083842 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.083694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.083842 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.083721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-home\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.083842 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.083799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-dshm\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.083842 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.083836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-model-cache\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.083983 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.083864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b6341c-d538-4cf5-a311-1d1735f601f2-tls-certs\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.083983 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.083902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.083983 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.083931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6341c-d538-4cf5-a311-1d1735f601f2-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.184524 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.184491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-model-cache\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.184524 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.184526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b6341c-d538-4cf5-a311-1d1735f601f2-tls-certs\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.184803 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.184549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.184803 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.184566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6341c-d538-4cf5-a311-1d1735f601f2-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.184803 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.184587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5gt\" (UniqueName: \"kubernetes.io/projected/c2b6341c-d538-4cf5-a311-1d1735f601f2-kube-api-access-ks5gt\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.184803 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.184639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.185014 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.184793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-home\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.185014 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.184867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-dshm\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.185014 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.184973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-model-cache\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.185157 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.185060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.185157 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.185133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.185232 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.185224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-home\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.185409 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.185385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6341c-d538-4cf5-a311-1d1735f601f2-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.187047 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.187025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-dshm\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.187227 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.187209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b6341c-d538-4cf5-a311-1d1735f601f2-tls-certs\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.193415 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.193391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5gt\" (UniqueName: \"kubernetes.io/projected/c2b6341c-d538-4cf5-a311-1d1735f601f2-kube-api-access-ks5gt\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.339280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.339190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:42.476380 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:42.476354    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"]
Aug 04 18:28:42.478931 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:28:42.478899    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b6341c_d538_4cf5_a311_1d1735f601f2.slice/crio-b243e50dfb931c251a19001b353a482ee736d47ac839859f1c45d2b770c4ce65 WatchSource:0}: Error finding container b243e50dfb931c251a19001b353a482ee736d47ac839859f1c45d2b770c4ce65: Status 404 returned error can't find the container with id b243e50dfb931c251a19001b353a482ee736d47ac839859f1c45d2b770c4ce65
Aug 04 18:28:43.132356 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:43.132319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" event={"ID":"c2b6341c-d538-4cf5-a311-1d1735f601f2","Type":"ContainerStarted","Data":"98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b"}
Aug 04 18:28:43.132761 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:43.132378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" event={"ID":"c2b6341c-d538-4cf5-a311-1d1735f601f2","Type":"ContainerStarted","Data":"b243e50dfb931c251a19001b353a482ee736d47ac839859f1c45d2b770c4ce65"}
Aug 04 18:28:45.141481 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:45.141445    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2b6341c-d538-4cf5-a311-1d1735f601f2" containerID="98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b" exitCode=0
Aug 04 18:28:45.141958 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:45.141514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" event={"ID":"c2b6341c-d538-4cf5-a311-1d1735f601f2","Type":"ContainerDied","Data":"98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b"}
Aug 04 18:28:46.146927 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:46.146895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" event={"ID":"c2b6341c-d538-4cf5-a311-1d1735f601f2","Type":"ContainerStarted","Data":"7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7"}
Aug 04 18:28:46.169334 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:46.169285    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" podStartSLOduration=3.168903907 podStartE2EDuration="5.16927044s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.000366533s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:28:46.167279215 +0000 UTC m=+1734.101349114" watchObservedRunningTime="2026-08-04 18:28:46.16927044 +0000 UTC m=+1734.103340339"
Aug 04 18:28:46.340157 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:46.340117    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:52.339469 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:52.339422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:52.351984 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:52.351958    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:28:53.191536 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:28:53.191506    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:29:21.341890 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.341855    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"]
Aug 04 18:29:21.350474 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:29:21.350430    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:29:21.350474 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:29:21.350474 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:29:21.350474 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7" cmd=["/bin/sleep","15"]
Aug 04 18:29:21.350838 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.350486    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" message="" err=<
Aug 04 18:29:21.350838 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:29:21.350838 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:29:21.350838 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:29:21.350838 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:29:21.350498    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:29:21.350838 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:29:21.350838 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:29:21.350838 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" podUID="c2b6341c-d538-4cf5-a311-1d1735f601f2" containerName="main" containerID="cri-o://7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7"
Aug 04 18:29:21.350838 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.350533    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" podUID="c2b6341c-d538-4cf5-a311-1d1735f601f2" containerName="main" containerID="cri-o://7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7" gracePeriod=60
Aug 04 18:29:21.705097 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.705074    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:29:21.718342 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718319    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6341c-d538-4cf5-a311-1d1735f601f2-cabundle-cert\") pod \"c2b6341c-d538-4cf5-a311-1d1735f601f2\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") "
Aug 04 18:29:21.718454 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718353    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-model-cache\") pod \"c2b6341c-d538-4cf5-a311-1d1735f601f2\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") "
Aug 04 18:29:21.718454 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks5gt\" (UniqueName: \"kubernetes.io/projected/c2b6341c-d538-4cf5-a311-1d1735f601f2-kube-api-access-ks5gt\") pod \"c2b6341c-d538-4cf5-a311-1d1735f601f2\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") "
Aug 04 18:29:21.718454 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-dshm\") pod \"c2b6341c-d538-4cf5-a311-1d1735f601f2\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") "
Aug 04 18:29:21.718454 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-kserve-provision-location\") pod \"c2b6341c-d538-4cf5-a311-1d1735f601f2\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") "
Aug 04 18:29:21.718706 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718469    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-tmp-dir\") pod \"c2b6341c-d538-4cf5-a311-1d1735f601f2\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") "
Aug 04 18:29:21.718706 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-home\") pod \"c2b6341c-d538-4cf5-a311-1d1735f601f2\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") "
Aug 04 18:29:21.718706 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718523    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b6341c-d538-4cf5-a311-1d1735f601f2-tls-certs\") pod \"c2b6341c-d538-4cf5-a311-1d1735f601f2\" (UID: \"c2b6341c-d538-4cf5-a311-1d1735f601f2\") "
Aug 04 18:29:21.718706 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718676    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2b6341c-d538-4cf5-a311-1d1735f601f2" (UID: "c2b6341c-d538-4cf5-a311-1d1735f601f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.718926 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718768    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b6341c-d538-4cf5-a311-1d1735f601f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c2b6341c-d538-4cf5-a311-1d1735f601f2" (UID: "c2b6341c-d538-4cf5-a311-1d1735f601f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:29:21.718926 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2b6341c-d538-4cf5-a311-1d1735f601f2" (UID: "c2b6341c-d538-4cf5-a311-1d1735f601f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.718926 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.718878    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-home" (OuterVolumeSpecName: "home") pod "c2b6341c-d538-4cf5-a311-1d1735f601f2" (UID: "c2b6341c-d538-4cf5-a311-1d1735f601f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.720648 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.720581    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b6341c-d538-4cf5-a311-1d1735f601f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2b6341c-d538-4cf5-a311-1d1735f601f2" (UID: "c2b6341c-d538-4cf5-a311-1d1735f601f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:29:21.720747 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.720679    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-dshm" (OuterVolumeSpecName: "dshm") pod "c2b6341c-d538-4cf5-a311-1d1735f601f2" (UID: "c2b6341c-d538-4cf5-a311-1d1735f601f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.720747 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.720680    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b6341c-d538-4cf5-a311-1d1735f601f2-kube-api-access-ks5gt" (OuterVolumeSpecName: "kube-api-access-ks5gt") pod "c2b6341c-d538-4cf5-a311-1d1735f601f2" (UID: "c2b6341c-d538-4cf5-a311-1d1735f601f2"). InnerVolumeSpecName "kube-api-access-ks5gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:29:21.744379 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.744344    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2b6341c-d538-4cf5-a311-1d1735f601f2" (UID: "c2b6341c-d538-4cf5-a311-1d1735f601f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.819579 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.819550    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.819579 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.819577    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.819579 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.819586    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b6341c-d538-4cf5-a311-1d1735f601f2-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.819828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.819594    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6341c-d538-4cf5-a311-1d1735f601f2-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.819828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.819621    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.819828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.819630    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks5gt\" (UniqueName: \"kubernetes.io/projected/c2b6341c-d538-4cf5-a311-1d1735f601f2-kube-api-access-ks5gt\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.819828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.819639    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.819828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:21.819647    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b6341c-d538-4cf5-a311-1d1735f601f2-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:29:22.284870 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.284833    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2b6341c-d538-4cf5-a311-1d1735f601f2" containerID="7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7" exitCode=0
Aug 04 18:29:22.285049 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.284887    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"
Aug 04 18:29:22.285049 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.284903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" event={"ID":"c2b6341c-d538-4cf5-a311-1d1735f601f2","Type":"ContainerDied","Data":"7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7"}
Aug 04 18:29:22.285049 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.284956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf" event={"ID":"c2b6341c-d538-4cf5-a311-1d1735f601f2","Type":"ContainerDied","Data":"b243e50dfb931c251a19001b353a482ee736d47ac839859f1c45d2b770c4ce65"}
Aug 04 18:29:22.285049 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.284978    2581 scope.go:117] "RemoveContainer" containerID="7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7"
Aug 04 18:29:22.293273 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.293250    2581 scope.go:117] "RemoveContainer" containerID="98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b"
Aug 04 18:29:22.300680 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.300664    2581 scope.go:117] "RemoveContainer" containerID="7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7"
Aug 04 18:29:22.300911 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:29:22.300892    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7\": container with ID starting with 7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7 not found: ID does not exist" containerID="7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7"
Aug 04 18:29:22.300981 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.300924    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7"} err="failed to get container status \"7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7\": rpc error: code = NotFound desc = could not find container \"7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7\": container with ID starting with 7c2ebe1d5e53356c5b70ad25a9d30ac0d5d00714e87adc4989f8a1d7e1814cf7 not found: ID does not exist"
Aug 04 18:29:22.300981 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.300947    2581 scope.go:117] "RemoveContainer" containerID="98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b"
Aug 04 18:29:22.301159 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:29:22.301142    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b\": container with ID starting with 98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b not found: ID does not exist" containerID="98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b"
Aug 04 18:29:22.301200 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.301164    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b"} err="failed to get container status \"98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b\": rpc error: code = NotFound desc = could not find container \"98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b\": container with ID starting with 98ace386c1908e2c40d6a204ec86465eb0fd9cb21126c5b770ec275a54dd444b not found: ID does not exist"
Aug 04 18:29:22.306537 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.306515    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"]
Aug 04 18:29:22.308755 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.308736    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-c6pnf"]
Aug 04 18:29:22.545513 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:29:22.545429    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2b6341c-d538-4cf5-a311-1d1735f601f2" path="/var/lib/kubelet/pods/c2b6341c-d538-4cf5-a311-1d1735f601f2/volumes"
Aug 04 18:30:19.202108 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.202070    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"]
Aug 04 18:30:19.202628 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.202331    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b6341c-d538-4cf5-a311-1d1735f601f2" containerName="storage-initializer"
Aug 04 18:30:19.202628 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.202341    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b6341c-d538-4cf5-a311-1d1735f601f2" containerName="storage-initializer"
Aug 04 18:30:19.202628 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.202358    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b6341c-d538-4cf5-a311-1d1735f601f2" containerName="main"
Aug 04 18:30:19.202628 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.202365    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b6341c-d538-4cf5-a311-1d1735f601f2" containerName="main"
Aug 04 18:30:19.202628 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.202410    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b6341c-d538-4cf5-a311-1d1735f601f2" containerName="main"
Aug 04 18:30:19.205051 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.205027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.207758 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.207735    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Aug 04 18:30:19.208910 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.208876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Aug 04 18:30:19.208910 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.208879    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-5bxcp\""
Aug 04 18:30:19.209093 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.208920    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Aug 04 18:30:19.218689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.218666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"]
Aug 04 18:30:19.273205 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.273169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkz5\" (UniqueName: \"kubernetes.io/projected/24c4e5f8-6f61-40b1-9439-378c9134409d-kube-api-access-xtkz5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.273363 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.273214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.273363 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.273273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.273363 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.273305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.273363 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.273323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c4e5f8-6f61-40b1-9439-378c9134409d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.273363 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.273342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.373904 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.373863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.374100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.373918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.374100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.373957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.374100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.373983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c4e5f8-6f61-40b1-9439-378c9134409d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.374100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.374010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.374324 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.374156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkz5\" (UniqueName: \"kubernetes.io/projected/24c4e5f8-6f61-40b1-9439-378c9134409d-kube-api-access-xtkz5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.374418 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.374394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.374481 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.374426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.374481 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.374453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.376252 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.376230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.376361 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.376345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c4e5f8-6f61-40b1-9439-378c9134409d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.385457 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.385430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkz5\" (UniqueName: \"kubernetes.io/projected/24c4e5f8-6f61-40b1-9439-378c9134409d-kube-api-access-xtkz5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.516309 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.516211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:19.659888 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:19.659862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"]
Aug 04 18:30:19.661887 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:30:19.661860    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c4e5f8_6f61_40b1_9439_378c9134409d.slice/crio-9f389224469d3b35b8848189d3714f8a1279d23bb755309236034fb7ef2cdbe6 WatchSource:0}: Error finding container 9f389224469d3b35b8848189d3714f8a1279d23bb755309236034fb7ef2cdbe6: Status 404 returned error can't find the container with id 9f389224469d3b35b8848189d3714f8a1279d23bb755309236034fb7ef2cdbe6
Aug 04 18:30:20.500457 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:20.500423    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx" event={"ID":"24c4e5f8-6f61-40b1-9439-378c9134409d","Type":"ContainerStarted","Data":"e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d"}
Aug 04 18:30:20.500457 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:20.500474    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx" event={"ID":"24c4e5f8-6f61-40b1-9439-378c9134409d","Type":"ContainerStarted","Data":"9f389224469d3b35b8848189d3714f8a1279d23bb755309236034fb7ef2cdbe6"}
Aug 04 18:30:20.516685 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:20.516650    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:20.519954 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:20.519900    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx" podStartSLOduration=1.519880447 podStartE2EDuration="1.519880447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:30:20.518103593 +0000 UTC m=+1828.452173505" watchObservedRunningTime="2026-08-04 18:30:20.519880447 +0000 UTC m=+1828.453950347"
Aug 04 18:30:29.517071 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:29.517030    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:29.529814 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:29.529784    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:29.545349 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:29.545321    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:56.109456 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.109408    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"]
Aug 04 18:30:56.117998 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:30:56.117963    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:30:56.117998 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:30:56.117998 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:30:56.117998 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d" cmd=["/bin/sleep","15"]
Aug 04 18:30:56.118236 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.118015    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx" message="" err=<
Aug 04 18:30:56.118236 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:30:56.118236 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:30:56.118236 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:30:56.118236 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:30:56.118027    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:30:56.118236 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:30:56.118236 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:30:56.118236 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx" podUID="24c4e5f8-6f61-40b1-9439-378c9134409d" containerName="main" containerID="cri-o://e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d"
Aug 04 18:30:56.118236 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.118075    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx" podUID="24c4e5f8-6f61-40b1-9439-378c9134409d" containerName="main" containerID="cri-o://e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d" gracePeriod=60
Aug 04 18:30:56.468196 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.468171    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:56.577478 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-model-cache\") pod \"24c4e5f8-6f61-40b1-9439-378c9134409d\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") "
Aug 04 18:30:56.577698 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577485    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-tmp-dir\") pod \"24c4e5f8-6f61-40b1-9439-378c9134409d\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") "
Aug 04 18:30:56.577698 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c4e5f8-6f61-40b1-9439-378c9134409d-tls-certs\") pod \"24c4e5f8-6f61-40b1-9439-378c9134409d\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") "
Aug 04 18:30:56.577698 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577565    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-home\") pod \"24c4e5f8-6f61-40b1-9439-378c9134409d\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") "
Aug 04 18:30:56.577698 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577587    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtkz5\" (UniqueName: \"kubernetes.io/projected/24c4e5f8-6f61-40b1-9439-378c9134409d-kube-api-access-xtkz5\") pod \"24c4e5f8-6f61-40b1-9439-378c9134409d\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") "
Aug 04 18:30:56.577698 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577642    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-dshm\") pod \"24c4e5f8-6f61-40b1-9439-378c9134409d\" (UID: \"24c4e5f8-6f61-40b1-9439-378c9134409d\") "
Aug 04 18:30:56.577905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24c4e5f8-6f61-40b1-9439-378c9134409d" (UID: "24c4e5f8-6f61-40b1-9439-378c9134409d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:30:56.577905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577785    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-model-cache" (OuterVolumeSpecName: "model-cache") pod "24c4e5f8-6f61-40b1-9439-378c9134409d" (UID: "24c4e5f8-6f61-40b1-9439-378c9134409d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:30:56.577905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577794    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-home" (OuterVolumeSpecName: "home") pod "24c4e5f8-6f61-40b1-9439-378c9134409d" (UID: "24c4e5f8-6f61-40b1-9439-378c9134409d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:30:56.577905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577872    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:30:56.577905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577891    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:30:56.577905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.577903    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:30:56.579964 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.579941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-dshm" (OuterVolumeSpecName: "dshm") pod "24c4e5f8-6f61-40b1-9439-378c9134409d" (UID: "24c4e5f8-6f61-40b1-9439-378c9134409d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:30:56.579964 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.579940    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c4e5f8-6f61-40b1-9439-378c9134409d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24c4e5f8-6f61-40b1-9439-378c9134409d" (UID: "24c4e5f8-6f61-40b1-9439-378c9134409d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:30:56.580117 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.579979    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c4e5f8-6f61-40b1-9439-378c9134409d-kube-api-access-xtkz5" (OuterVolumeSpecName: "kube-api-access-xtkz5") pod "24c4e5f8-6f61-40b1-9439-378c9134409d" (UID: "24c4e5f8-6f61-40b1-9439-378c9134409d"). InnerVolumeSpecName "kube-api-access-xtkz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:30:56.636555 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.636469    2581 generic.go:356] "Generic (PLEG): container finished" podID="24c4e5f8-6f61-40b1-9439-378c9134409d" containerID="e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d" exitCode=0
Aug 04 18:30:56.636555 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.636513    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"
Aug 04 18:30:56.636555 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.636544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx" event={"ID":"24c4e5f8-6f61-40b1-9439-378c9134409d","Type":"ContainerDied","Data":"e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d"}
Aug 04 18:30:56.636787 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.636614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx" event={"ID":"24c4e5f8-6f61-40b1-9439-378c9134409d","Type":"ContainerDied","Data":"9f389224469d3b35b8848189d3714f8a1279d23bb755309236034fb7ef2cdbe6"}
Aug 04 18:30:56.636787 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.636632    2581 scope.go:117] "RemoveContainer" containerID="e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d"
Aug 04 18:30:56.645173 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.645156    2581 scope.go:117] "RemoveContainer" containerID="e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d"
Aug 04 18:30:56.645450 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:30:56.645425    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d\": container with ID starting with e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d not found: ID does not exist" containerID="e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d"
Aug 04 18:30:56.645522 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.645461    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d"} err="failed to get container status \"e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d\": rpc error: code = NotFound desc = could not find container \"e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d\": container with ID starting with e31f210d3716e3efb766bc3475d06f97b0ddaa6accf7604956a7bf8755d80b6d not found: ID does not exist"
Aug 04 18:30:56.664958 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.664930    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"]
Aug 04 18:30:56.671393 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.671366    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rldpx"]
Aug 04 18:30:56.679227 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.679208    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c4e5f8-6f61-40b1-9439-378c9134409d-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:30:56.679317 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.679230    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtkz5\" (UniqueName: \"kubernetes.io/projected/24c4e5f8-6f61-40b1-9439-378c9134409d-kube-api-access-xtkz5\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:30:56.679317 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:56.679243    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c4e5f8-6f61-40b1-9439-378c9134409d-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:30:58.545037 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:30:58.545004    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c4e5f8-6f61-40b1-9439-378c9134409d" path="/var/lib/kubelet/pods/24c4e5f8-6f61-40b1-9439-378c9134409d/volumes"
Aug 04 18:32:21.403863 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.403831    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"]
Aug 04 18:32:21.404250 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.404090    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c4e5f8-6f61-40b1-9439-378c9134409d" containerName="main"
Aug 04 18:32:21.404250 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.404100    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c4e5f8-6f61-40b1-9439-378c9134409d" containerName="main"
Aug 04 18:32:21.404250 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.404168    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c4e5f8-6f61-40b1-9439-378c9134409d" containerName="main"
Aug 04 18:32:21.406910 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.406894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.411242 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.411219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Aug 04 18:32:21.411242 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.411230    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-g5ksp\""
Aug 04 18:32:21.411441 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.411255    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Aug 04 18:32:21.411441 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.411327    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Aug 04 18:32:21.421443 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.421423    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"]
Aug 04 18:32:21.558704 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.558666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkn7\" (UniqueName: \"kubernetes.io/projected/533fe940-2d16-48ff-b3b4-59857f0f2ef9-kube-api-access-hhkn7\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.558880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.558713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.558880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.558816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.558880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.558855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.558880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.558873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.559068 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.558906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.659390 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.659294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.659390 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.659341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.659390 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.659366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.659671 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.659394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.659671 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.659444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkn7\" (UniqueName: \"kubernetes.io/projected/533fe940-2d16-48ff-b3b4-59857f0f2ef9-kube-api-access-hhkn7\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.659671 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.659478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.659834 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.659813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.659915 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.659854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.660000 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.659984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.661777 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.661755    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.662187 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.662166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.680934 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.680906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkn7\" (UniqueName: \"kubernetes.io/projected/533fe940-2d16-48ff-b3b4-59857f0f2ef9-kube-api-access-hhkn7\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.716412 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.716383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:21.847344 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.847160    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"]
Aug 04 18:32:21.850088 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:32:21.850062    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533fe940_2d16_48ff_b3b4_59857f0f2ef9.slice/crio-3e9302484803d0e5a11b21fa07f6427746cbcd349bd9325e71e911214f850d0b WatchSource:0}: Error finding container 3e9302484803d0e5a11b21fa07f6427746cbcd349bd9325e71e911214f850d0b: Status 404 returned error can't find the container with id 3e9302484803d0e5a11b21fa07f6427746cbcd349bd9325e71e911214f850d0b
Aug 04 18:32:21.944129 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.944050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q" event={"ID":"533fe940-2d16-48ff-b3b4-59857f0f2ef9","Type":"ContainerStarted","Data":"7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a"}
Aug 04 18:32:21.944129 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.944108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q" event={"ID":"533fe940-2d16-48ff-b3b4-59857f0f2ef9","Type":"ContainerStarted","Data":"3e9302484803d0e5a11b21fa07f6427746cbcd349bd9325e71e911214f850d0b"}
Aug 04 18:32:21.962672 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:21.962598    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q" podStartSLOduration=0.962584187 podStartE2EDuration="962.584187ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:32:21.96203287 +0000 UTC m=+1949.896102769" watchObservedRunningTime="2026-08-04 18:32:21.962584187 +0000 UTC m=+1949.896654085"
Aug 04 18:32:22.717175 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:22.717134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:31.717009 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:31.716975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:31.729335 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:31.729310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:32:31.994278 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:32:31.994246    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:33:15.599061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.598983    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"]
Aug 04 18:33:15.602061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.602045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.606164 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-home\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.606164 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a45ced-760e-4608-b544-8427b269713d-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.606164 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606152    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Aug 04 18:33:15.606404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606179    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Aug 04 18:33:15.606404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-dshm\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.606404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Aug 04 18:33:15.606404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606283    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-4zcxc\""
Aug 04 18:33:15.606404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.606404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4f7l\" (UniqueName: \"kubernetes.io/projected/10a45ced-760e-4608-b544-8427b269713d-kube-api-access-c4f7l\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.606404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.606362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-model-cache\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.618868 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.618849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"]
Aug 04 18:33:15.707045 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.707010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a45ced-760e-4608-b544-8427b269713d-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.707045 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.707044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-dshm\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.707293 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.707082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.707293 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.707118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4f7l\" (UniqueName: \"kubernetes.io/projected/10a45ced-760e-4608-b544-8427b269713d-kube-api-access-c4f7l\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.707293 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.707260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-model-cache\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.707450 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.707366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-home\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.707689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.707657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.707689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.707671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-model-cache\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.707689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.707682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-home\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.709408 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.709377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-dshm\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.709824 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.709798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a45ced-760e-4608-b544-8427b269713d-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.716140 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.716115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4f7l\" (UniqueName: \"kubernetes.io/projected/10a45ced-760e-4608-b544-8427b269713d-kube-api-access-c4f7l\") pod \"leave-v2-kserve-556b4dddf9-cfq7l\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:15.912040 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:15.912000    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:16.041544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:16.041516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"]
Aug 04 18:33:16.042042 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:33:16.042017    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a45ced_760e_4608_b544_8427b269713d.slice/crio-86c56ae6534ebf1cda2c01a440c59fa0bcdbfde9da7d8bf93de5c720cf58cf69 WatchSource:0}: Error finding container 86c56ae6534ebf1cda2c01a440c59fa0bcdbfde9da7d8bf93de5c720cf58cf69: Status 404 returned error can't find the container with id 86c56ae6534ebf1cda2c01a440c59fa0bcdbfde9da7d8bf93de5c720cf58cf69
Aug 04 18:33:16.152922 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:16.152883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l" event={"ID":"10a45ced-760e-4608-b544-8427b269713d","Type":"ContainerStarted","Data":"be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce"}
Aug 04 18:33:16.153088 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:16.152944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l" event={"ID":"10a45ced-760e-4608-b544-8427b269713d","Type":"ContainerStarted","Data":"86c56ae6534ebf1cda2c01a440c59fa0bcdbfde9da7d8bf93de5c720cf58cf69"}
Aug 04 18:33:16.175783 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:16.175684    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l" podStartSLOduration=1.175667938 podStartE2EDuration="1.175667938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:33:16.17425596 +0000 UTC m=+2004.108325859" watchObservedRunningTime="2026-08-04 18:33:16.175667938 +0000 UTC m=+2004.109737837"
Aug 04 18:33:16.912770 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:16.912714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:25.913062 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:25.913016    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:25.925561 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:25.925531    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:26.203185 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:26.203099    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:33:53.212153 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.212120    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"]
Aug 04 18:33:53.221025 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:33:53.220994    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:33:53.221025 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:33:53.221025 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:33:53.221025 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a" cmd=["/bin/sleep","15"]
Aug 04 18:33:53.221336 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.221039    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q" message="" err=<
Aug 04 18:33:53.221336 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:33:53.221336 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:33:53.221336 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:33:53.221336 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:33:53.221051    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:33:53.221336 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:33:53.221336 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:33:53.221336 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q" podUID="533fe940-2d16-48ff-b3b4-59857f0f2ef9" containerName="main" containerID="cri-o://7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a"
Aug 04 18:33:53.221336 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.221098    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q" podUID="533fe940-2d16-48ff-b3b4-59857f0f2ef9" containerName="main" containerID="cri-o://7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a" gracePeriod=60
Aug 04 18:33:53.561920 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.561899    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:33:53.723356 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.723311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhkn7\" (UniqueName: \"kubernetes.io/projected/533fe940-2d16-48ff-b3b4-59857f0f2ef9-kube-api-access-hhkn7\") pod \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") "
Aug 04 18:33:53.723356 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.723355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-dshm\") pod \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") "
Aug 04 18:33:53.723657 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.723381    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-home\") pod \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") "
Aug 04 18:33:53.723657 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.723417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tls-certs\") pod \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") "
Aug 04 18:33:53.723657 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.723438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tmp-dir\") pod \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") "
Aug 04 18:33:53.723657 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.723462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-model-cache\") pod \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\" (UID: \"533fe940-2d16-48ff-b3b4-59857f0f2ef9\") "
Aug 04 18:33:53.723891 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.723797    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-home" (OuterVolumeSpecName: "home") pod "533fe940-2d16-48ff-b3b4-59857f0f2ef9" (UID: "533fe940-2d16-48ff-b3b4-59857f0f2ef9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:33:53.723891 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.723782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "533fe940-2d16-48ff-b3b4-59857f0f2ef9" (UID: "533fe940-2d16-48ff-b3b4-59857f0f2ef9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:33:53.723891 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.723844    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-model-cache" (OuterVolumeSpecName: "model-cache") pod "533fe940-2d16-48ff-b3b4-59857f0f2ef9" (UID: "533fe940-2d16-48ff-b3b4-59857f0f2ef9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:33:53.725697 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.725672    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-dshm" (OuterVolumeSpecName: "dshm") pod "533fe940-2d16-48ff-b3b4-59857f0f2ef9" (UID: "533fe940-2d16-48ff-b3b4-59857f0f2ef9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:33:53.725820 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.725709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "533fe940-2d16-48ff-b3b4-59857f0f2ef9" (UID: "533fe940-2d16-48ff-b3b4-59857f0f2ef9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:33:53.725820 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.725795    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533fe940-2d16-48ff-b3b4-59857f0f2ef9-kube-api-access-hhkn7" (OuterVolumeSpecName: "kube-api-access-hhkn7") pod "533fe940-2d16-48ff-b3b4-59857f0f2ef9" (UID: "533fe940-2d16-48ff-b3b4-59857f0f2ef9"). InnerVolumeSpecName "kube-api-access-hhkn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:33:53.824061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.823985    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:33:53.824061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.824010    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:33:53.824061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.824018    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:33:53.824061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.824027    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhkn7\" (UniqueName: \"kubernetes.io/projected/533fe940-2d16-48ff-b3b4-59857f0f2ef9-kube-api-access-hhkn7\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:33:53.824061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.824036    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:33:53.824061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:53.824044    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533fe940-2d16-48ff-b3b4-59857f0f2ef9-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:33:54.301347 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.301312    2581 generic.go:356] "Generic (PLEG): container finished" podID="533fe940-2d16-48ff-b3b4-59857f0f2ef9" containerID="7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a" exitCode=0
Aug 04 18:33:54.301799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.301368    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"
Aug 04 18:33:54.301799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.301395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q" event={"ID":"533fe940-2d16-48ff-b3b4-59857f0f2ef9","Type":"ContainerDied","Data":"7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a"}
Aug 04 18:33:54.301799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.301452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q" event={"ID":"533fe940-2d16-48ff-b3b4-59857f0f2ef9","Type":"ContainerDied","Data":"3e9302484803d0e5a11b21fa07f6427746cbcd349bd9325e71e911214f850d0b"}
Aug 04 18:33:54.301799 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.301479    2581 scope.go:117] "RemoveContainer" containerID="7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a"
Aug 04 18:33:54.310531 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.310514    2581 scope.go:117] "RemoveContainer" containerID="7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a"
Aug 04 18:33:54.310854 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:33:54.310832    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a\": container with ID starting with 7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a not found: ID does not exist" containerID="7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a"
Aug 04 18:33:54.310912 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.310861    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a"} err="failed to get container status \"7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a\": rpc error: code = NotFound desc = could not find container \"7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a\": container with ID starting with 7c83b2ee1c5ad9ae2e8ebe4ec7fd4c6244deea99931f6eab1914a2c834a4248a not found: ID does not exist"
Aug 04 18:33:54.323914 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.323884    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"]
Aug 04 18:33:54.330573 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.330552    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ktf2q"]
Aug 04 18:33:54.545440 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:33:54.545408    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533fe940-2d16-48ff-b3b4-59857f0f2ef9" path="/var/lib/kubelet/pods/533fe940-2d16-48ff-b3b4-59857f0f2ef9/volumes"
Aug 04 18:34:01.330657 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:01.330627    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"]
Aug 04 18:34:16.348700 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.348590    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l" podUID="10a45ced-760e-4608-b544-8427b269713d" containerName="main" containerID="cri-o://be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce" gracePeriod=45
Aug 04 18:34:16.593549 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.593523    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:34:16.611548 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.611488    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4f7l\" (UniqueName: \"kubernetes.io/projected/10a45ced-760e-4608-b544-8427b269713d-kube-api-access-c4f7l\") pod \"10a45ced-760e-4608-b544-8427b269713d\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") "
Aug 04 18:34:16.611548 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.611521    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-dshm\") pod \"10a45ced-760e-4608-b544-8427b269713d\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") "
Aug 04 18:34:16.611548 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.611543    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a45ced-760e-4608-b544-8427b269713d-tls-certs\") pod \"10a45ced-760e-4608-b544-8427b269713d\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") "
Aug 04 18:34:16.611812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.611561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-home\") pod \"10a45ced-760e-4608-b544-8427b269713d\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") "
Aug 04 18:34:16.611812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.611588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-tmp-dir\") pod \"10a45ced-760e-4608-b544-8427b269713d\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") "
Aug 04 18:34:16.611812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.611643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-model-cache\") pod \"10a45ced-760e-4608-b544-8427b269713d\" (UID: \"10a45ced-760e-4608-b544-8427b269713d\") "
Aug 04 18:34:16.611971 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.611850    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-home" (OuterVolumeSpecName: "home") pod "10a45ced-760e-4608-b544-8427b269713d" (UID: "10a45ced-760e-4608-b544-8427b269713d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:16.611971 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.611916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10a45ced-760e-4608-b544-8427b269713d" (UID: "10a45ced-760e-4608-b544-8427b269713d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:16.611971 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.611930    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-model-cache" (OuterVolumeSpecName: "model-cache") pod "10a45ced-760e-4608-b544-8427b269713d" (UID: "10a45ced-760e-4608-b544-8427b269713d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:16.613879 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.613857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a45ced-760e-4608-b544-8427b269713d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10a45ced-760e-4608-b544-8427b269713d" (UID: "10a45ced-760e-4608-b544-8427b269713d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:34:16.613879 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.613870    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-dshm" (OuterVolumeSpecName: "dshm") pod "10a45ced-760e-4608-b544-8427b269713d" (UID: "10a45ced-760e-4608-b544-8427b269713d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:16.613987 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.613884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a45ced-760e-4608-b544-8427b269713d-kube-api-access-c4f7l" (OuterVolumeSpecName: "kube-api-access-c4f7l") pod "10a45ced-760e-4608-b544-8427b269713d" (UID: "10a45ced-760e-4608-b544-8427b269713d"). InnerVolumeSpecName "kube-api-access-c4f7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:34:16.712395 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.712358    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4f7l\" (UniqueName: \"kubernetes.io/projected/10a45ced-760e-4608-b544-8427b269713d-kube-api-access-c4f7l\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:34:16.712395 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.712389    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:34:16.712395 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.712399    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a45ced-760e-4608-b544-8427b269713d-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:34:16.712672 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.712409    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:34:16.712672 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.712417    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:34:16.712672 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:16.712424    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a45ced-760e-4608-b544-8427b269713d-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:34:17.389666 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:17.389623    2581 generic.go:356] "Generic (PLEG): container finished" podID="10a45ced-760e-4608-b544-8427b269713d" containerID="be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce" exitCode=0
Aug 04 18:34:17.389666 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:17.389635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l" event={"ID":"10a45ced-760e-4608-b544-8427b269713d","Type":"ContainerDied","Data":"be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce"}
Aug 04 18:34:17.390218 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:17.389677    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"
Aug 04 18:34:17.390218 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:17.389749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l" event={"ID":"10a45ced-760e-4608-b544-8427b269713d","Type":"ContainerDied","Data":"86c56ae6534ebf1cda2c01a440c59fa0bcdbfde9da7d8bf93de5c720cf58cf69"}
Aug 04 18:34:17.390218 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:17.389778    2581 scope.go:117] "RemoveContainer" containerID="be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce"
Aug 04 18:34:17.397592 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:17.397569    2581 scope.go:117] "RemoveContainer" containerID="be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce"
Aug 04 18:34:17.397931 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:34:17.397866    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce\": container with ID starting with be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce not found: ID does not exist" containerID="be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce"
Aug 04 18:34:17.398002 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:17.397927    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce"} err="failed to get container status \"be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce\": rpc error: code = NotFound desc = could not find container \"be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce\": container with ID starting with be99ace8406c37c90c11ecda6ac41bd82b229094019aeb6bd6d7348878e57dce not found: ID does not exist"
Aug 04 18:34:17.411780 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:17.411754    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"]
Aug 04 18:34:17.415520 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:17.415497    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-cfq7l"]
Aug 04 18:34:18.545611 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:34:18.545572    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a45ced-760e-4608-b544-8427b269713d" path="/var/lib/kubelet/pods/10a45ced-760e-4608-b544-8427b269713d/volumes"
Aug 04 18:36:53.896970 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.896938    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"]
Aug 04 18:36:53.897371 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.897199    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533fe940-2d16-48ff-b3b4-59857f0f2ef9" containerName="main"
Aug 04 18:36:53.897371 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.897210    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="533fe940-2d16-48ff-b3b4-59857f0f2ef9" containerName="main"
Aug 04 18:36:53.897371 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.897231    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a45ced-760e-4608-b544-8427b269713d" containerName="main"
Aug 04 18:36:53.897371 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.897237    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a45ced-760e-4608-b544-8427b269713d" containerName="main"
Aug 04 18:36:53.897371 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.897278    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="533fe940-2d16-48ff-b3b4-59857f0f2ef9" containerName="main"
Aug 04 18:36:53.897371 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.897286    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a45ced-760e-4608-b544-8427b269713d" containerName="main"
Aug 04 18:36:53.899883 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.899866    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:53.903968 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.903944    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Aug 04 18:36:53.904100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.903987    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-7zpwf\""
Aug 04 18:36:53.904100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.904059    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Aug 04 18:36:53.904100 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.904082    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Aug 04 18:36:53.912179 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.912158    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"]
Aug 04 18:36:53.997559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.997513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:53.997559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.997559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:53.997818 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.997651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq75w\" (UniqueName: \"kubernetes.io/projected/29f5c779-4ea7-4122-a523-ce8aae3f93a0-kube-api-access-cq75w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:53.997818 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.997692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:53.997818 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.997710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:53.997818 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:53.997727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.098354 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.098311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.098354 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.098357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.098646 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.098387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.098646 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.098427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.098646 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.098452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.098646 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.098501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq75w\" (UniqueName: \"kubernetes.io/projected/29f5c779-4ea7-4122-a523-ce8aae3f93a0-kube-api-access-cq75w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.098936 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.098911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.098997 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.098937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.098997 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.098961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.101221 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.101195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.101463 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.101445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.108585 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.108556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq75w\" (UniqueName: \"kubernetes.io/projected/29f5c779-4ea7-4122-a523-ce8aae3f93a0-kube-api-access-cq75w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.210465 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.210385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:36:54.339869 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.339843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"]
Aug 04 18:36:54.342792 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:36:54.342768    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f5c779_4ea7_4122_a523_ce8aae3f93a0.slice/crio-86d78de6ca5f083fa50422f7efc6dc61ce0315f27f8d8518f62d980322cbd602 WatchSource:0}: Error finding container 86d78de6ca5f083fa50422f7efc6dc61ce0315f27f8d8518f62d980322cbd602: Status 404 returned error can't find the container with id 86d78de6ca5f083fa50422f7efc6dc61ce0315f27f8d8518f62d980322cbd602
Aug 04 18:36:54.965281 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.965239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6" event={"ID":"29f5c779-4ea7-4122-a523-ce8aae3f93a0","Type":"ContainerStarted","Data":"cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533"}
Aug 04 18:36:54.965728 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.965292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6" event={"ID":"29f5c779-4ea7-4122-a523-ce8aae3f93a0","Type":"ContainerStarted","Data":"86d78de6ca5f083fa50422f7efc6dc61ce0315f27f8d8518f62d980322cbd602"}
Aug 04 18:36:54.985081 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:54.985034    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6" podStartSLOduration=1.9850206350000001 podStartE2EDuration="1.985020635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:36:54.983793262 +0000 UTC m=+2222.917863161" watchObservedRunningTime="2026-08-04 18:36:54.985020635 +0000 UTC m=+2222.919090776"
Aug 04 18:36:55.211464 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:36:55.211413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:37:04.210663 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:04.210627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:37:04.223156 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:04.223127    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:37:05.014663 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:05.014631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:37:21.506787 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.506750    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"]
Aug 04 18:37:21.510024 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.510006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.512813 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.512785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-bpt7h\""
Aug 04 18:37:21.513958 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.513938    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Aug 04 18:37:21.514096 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.514063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Aug 04 18:37:21.514397 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.514380    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Aug 04 18:37:21.521115 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.521091    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"]
Aug 04 18:37:21.606088 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.606055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-tmp-dir\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.606247 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.606103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-dshm\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.606247 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.606142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-home\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.606247 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.606178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3440818-4ee0-477f-b822-dad5fdb33d8a-tls-certs\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.606247 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.606203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-model-cache\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.606247 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.606222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6ql\" (UniqueName: \"kubernetes.io/projected/d3440818-4ee0-477f-b822-dad5fdb33d8a-kube-api-access-5k6ql\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.707089 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.707050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3440818-4ee0-477f-b822-dad5fdb33d8a-tls-certs\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.707089 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.707089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-model-cache\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.707344 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.707114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6ql\" (UniqueName: \"kubernetes.io/projected/d3440818-4ee0-477f-b822-dad5fdb33d8a-kube-api-access-5k6ql\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.707344 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.707136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-tmp-dir\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.707344 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.707175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-dshm\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.707344 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.707208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-home\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.707544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.707485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-model-cache\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.707629 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.707546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-home\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.707629 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.707589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-tmp-dir\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.709544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.709516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-dshm\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.709715 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.709700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3440818-4ee0-477f-b822-dad5fdb33d8a-tls-certs\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.715980 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.715947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6ql\" (UniqueName: \"kubernetes.io/projected/d3440818-4ee0-477f-b822-dad5fdb33d8a-kube-api-access-5k6ql\") pod \"late-v2-kserve-86766546f-9lh25\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.821138 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.821047    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:21.950253 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:21.950227    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"]
Aug 04 18:37:21.952712 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:37:21.952681    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3440818_4ee0_477f_b822_dad5fdb33d8a.slice/crio-299234b84aaa045da6258c41f84b7361587b2a30a6a468bab890b562900c19f3 WatchSource:0}: Error finding container 299234b84aaa045da6258c41f84b7361587b2a30a6a468bab890b562900c19f3: Status 404 returned error can't find the container with id 299234b84aaa045da6258c41f84b7361587b2a30a6a468bab890b562900c19f3
Aug 04 18:37:22.071420 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:22.071329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25" event={"ID":"d3440818-4ee0-477f-b822-dad5fdb33d8a","Type":"ContainerStarted","Data":"7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1"}
Aug 04 18:37:22.071420 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:22.071390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25" event={"ID":"d3440818-4ee0-477f-b822-dad5fdb33d8a","Type":"ContainerStarted","Data":"299234b84aaa045da6258c41f84b7361587b2a30a6a468bab890b562900c19f3"}
Aug 04 18:37:22.821363 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:22.821327    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:29.363686 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.363625    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25" podStartSLOduration=8.363585653 podStartE2EDuration="8.363585653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:37:22.09893098 +0000 UTC m=+2250.033000878" watchObservedRunningTime="2026-08-04 18:37:29.363585653 +0000 UTC m=+2257.297655553"
Aug 04 18:37:29.364742 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.364721    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"]
Aug 04 18:37:29.374499 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:37:29.374467    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:37:29.374499 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:37:29.374499 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:37:29.374499 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533" cmd=["/bin/sleep","15"]
Aug 04 18:37:29.374816 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.374518    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6" message="" err=<
Aug 04 18:37:29.374816 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:37:29.374816 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:37:29.374816 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:37:29.374816 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:37:29.374532    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:37:29.374816 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:37:29.374816 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:37:29.374816 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6" podUID="29f5c779-4ea7-4122-a523-ce8aae3f93a0" containerName="main" containerID="cri-o://cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533"
Aug 04 18:37:29.374816 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.374574    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6" podUID="29f5c779-4ea7-4122-a523-ce8aae3f93a0" containerName="main" containerID="cri-o://cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533" gracePeriod=60
Aug 04 18:37:29.724998 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.724971    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:37:29.776407 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.776368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq75w\" (UniqueName: \"kubernetes.io/projected/29f5c779-4ea7-4122-a523-ce8aae3f93a0-kube-api-access-cq75w\") pod \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") "
Aug 04 18:37:29.776553 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.776426    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-home\") pod \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") "
Aug 04 18:37:29.776553 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.776489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-dshm\") pod \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") "
Aug 04 18:37:29.776553 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.776529    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tmp-dir\") pod \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") "
Aug 04 18:37:29.776744 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.776555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tls-certs\") pod \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") "
Aug 04 18:37:29.776744 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.776628    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-model-cache\") pod \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\" (UID: \"29f5c779-4ea7-4122-a523-ce8aae3f93a0\") "
Aug 04 18:37:29.777245 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.776918    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-home" (OuterVolumeSpecName: "home") pod "29f5c779-4ea7-4122-a523-ce8aae3f93a0" (UID: "29f5c779-4ea7-4122-a523-ce8aae3f93a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:29.777245 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.777063    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "29f5c779-4ea7-4122-a523-ce8aae3f93a0" (UID: "29f5c779-4ea7-4122-a523-ce8aae3f93a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:29.777245 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.777100    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29f5c779-4ea7-4122-a523-ce8aae3f93a0" (UID: "29f5c779-4ea7-4122-a523-ce8aae3f93a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:29.778905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.778867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-dshm" (OuterVolumeSpecName: "dshm") pod "29f5c779-4ea7-4122-a523-ce8aae3f93a0" (UID: "29f5c779-4ea7-4122-a523-ce8aae3f93a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:29.779188 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.779150    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f5c779-4ea7-4122-a523-ce8aae3f93a0-kube-api-access-cq75w" (OuterVolumeSpecName: "kube-api-access-cq75w") pod "29f5c779-4ea7-4122-a523-ce8aae3f93a0" (UID: "29f5c779-4ea7-4122-a523-ce8aae3f93a0"). InnerVolumeSpecName "kube-api-access-cq75w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:37:29.779299 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.779249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29f5c779-4ea7-4122-a523-ce8aae3f93a0" (UID: "29f5c779-4ea7-4122-a523-ce8aae3f93a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:37:29.877840 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.877799    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:37:29.877840 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.877831    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:37:29.877840 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.877840    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f5c779-4ea7-4122-a523-ce8aae3f93a0-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:37:29.877840 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.877848    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:37:29.878105 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.877856    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq75w\" (UniqueName: \"kubernetes.io/projected/29f5c779-4ea7-4122-a523-ce8aae3f93a0-kube-api-access-cq75w\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:37:29.878105 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:29.877867    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f5c779-4ea7-4122-a523-ce8aae3f93a0-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:37:30.101110 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.101023    2581 generic.go:356] "Generic (PLEG): container finished" podID="29f5c779-4ea7-4122-a523-ce8aae3f93a0" containerID="cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533" exitCode=0
Aug 04 18:37:30.101110 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.101057    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"
Aug 04 18:37:30.101110 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.101100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6" event={"ID":"29f5c779-4ea7-4122-a523-ce8aae3f93a0","Type":"ContainerDied","Data":"cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533"}
Aug 04 18:37:30.101339 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.101147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6" event={"ID":"29f5c779-4ea7-4122-a523-ce8aae3f93a0","Type":"ContainerDied","Data":"86d78de6ca5f083fa50422f7efc6dc61ce0315f27f8d8518f62d980322cbd602"}
Aug 04 18:37:30.101339 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.101170    2581 scope.go:117] "RemoveContainer" containerID="cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533"
Aug 04 18:37:30.109829 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.109813    2581 scope.go:117] "RemoveContainer" containerID="cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533"
Aug 04 18:37:30.110048 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:37:30.110028    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533\": container with ID starting with cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533 not found: ID does not exist" containerID="cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533"
Aug 04 18:37:30.110112 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.110061    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533"} err="failed to get container status \"cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533\": rpc error: code = NotFound desc = could not find container \"cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533\": container with ID starting with cd89d2f20ec350b6a84922fb0c745e56b8f4ebe2d4b7315907ea3614d3e0f533 not found: ID does not exist"
Aug 04 18:37:30.122058 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.122034    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"]
Aug 04 18:37:30.124210 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.124189    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbgg6"]
Aug 04 18:37:30.550178 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:30.546068    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29f5c779-4ea7-4122-a523-ce8aae3f93a0" path="/var/lib/kubelet/pods/29f5c779-4ea7-4122-a523-ce8aae3f93a0/volumes"
Aug 04 18:37:31.821185 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:31.821160    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:31.834429 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:31.834406    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:37:32.121223 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:37:32.121195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:38:03.833129 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:03.833093    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"]
Aug 04 18:38:18.851728 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:18.851669    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25" podUID="d3440818-4ee0-477f-b822-dad5fdb33d8a" containerName="main" containerID="cri-o://7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1" gracePeriod=45
Aug 04 18:38:19.115892 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.115868    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:38:19.173841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.173814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-home\") pod \"d3440818-4ee0-477f-b822-dad5fdb33d8a\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") "
Aug 04 18:38:19.174031 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.173850    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-dshm\") pod \"d3440818-4ee0-477f-b822-dad5fdb33d8a\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") "
Aug 04 18:38:19.174031 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.173866    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3440818-4ee0-477f-b822-dad5fdb33d8a-tls-certs\") pod \"d3440818-4ee0-477f-b822-dad5fdb33d8a\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") "
Aug 04 18:38:19.174031 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.173886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-tmp-dir\") pod \"d3440818-4ee0-477f-b822-dad5fdb33d8a\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") "
Aug 04 18:38:19.174031 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.173925    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k6ql\" (UniqueName: \"kubernetes.io/projected/d3440818-4ee0-477f-b822-dad5fdb33d8a-kube-api-access-5k6ql\") pod \"d3440818-4ee0-477f-b822-dad5fdb33d8a\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") "
Aug 04 18:38:19.174031 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.174003    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-model-cache\") pod \"d3440818-4ee0-477f-b822-dad5fdb33d8a\" (UID: \"d3440818-4ee0-477f-b822-dad5fdb33d8a\") "
Aug 04 18:38:19.174310 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.174146    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-home" (OuterVolumeSpecName: "home") pod "d3440818-4ee0-477f-b822-dad5fdb33d8a" (UID: "d3440818-4ee0-477f-b822-dad5fdb33d8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:19.174310 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.174158    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3440818-4ee0-477f-b822-dad5fdb33d8a" (UID: "d3440818-4ee0-477f-b822-dad5fdb33d8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:19.174310 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.174244    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.174310 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.174262    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.174310 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.174253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3440818-4ee0-477f-b822-dad5fdb33d8a" (UID: "d3440818-4ee0-477f-b822-dad5fdb33d8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:19.176258 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.176230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-dshm" (OuterVolumeSpecName: "dshm") pod "d3440818-4ee0-477f-b822-dad5fdb33d8a" (UID: "d3440818-4ee0-477f-b822-dad5fdb33d8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:19.176258 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.176236    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3440818-4ee0-477f-b822-dad5fdb33d8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3440818-4ee0-477f-b822-dad5fdb33d8a" (UID: "d3440818-4ee0-477f-b822-dad5fdb33d8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:38:19.176258 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.176237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3440818-4ee0-477f-b822-dad5fdb33d8a-kube-api-access-5k6ql" (OuterVolumeSpecName: "kube-api-access-5k6ql") pod "d3440818-4ee0-477f-b822-dad5fdb33d8a" (UID: "d3440818-4ee0-477f-b822-dad5fdb33d8a"). InnerVolumeSpecName "kube-api-access-5k6ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:38:19.275373 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.275339    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.275373 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.275366    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3440818-4ee0-477f-b822-dad5fdb33d8a-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.275373 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.275375    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3440818-4ee0-477f-b822-dad5fdb33d8a-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.275373 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.275384    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6ql\" (UniqueName: \"kubernetes.io/projected/d3440818-4ee0-477f-b822-dad5fdb33d8a-kube-api-access-5k6ql\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.280226 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.280196    2581 generic.go:356] "Generic (PLEG): container finished" podID="d3440818-4ee0-477f-b822-dad5fdb33d8a" containerID="7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1" exitCode=0
Aug 04 18:38:19.280346 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.280246    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"
Aug 04 18:38:19.280346 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.280279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25" event={"ID":"d3440818-4ee0-477f-b822-dad5fdb33d8a","Type":"ContainerDied","Data":"7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1"}
Aug 04 18:38:19.280346 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.280336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25" event={"ID":"d3440818-4ee0-477f-b822-dad5fdb33d8a","Type":"ContainerDied","Data":"299234b84aaa045da6258c41f84b7361587b2a30a6a468bab890b562900c19f3"}
Aug 04 18:38:19.280455 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.280359    2581 scope.go:117] "RemoveContainer" containerID="7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1"
Aug 04 18:38:19.288618 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.288591    2581 scope.go:117] "RemoveContainer" containerID="7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1"
Aug 04 18:38:19.288888 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:38:19.288867    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1\": container with ID starting with 7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1 not found: ID does not exist" containerID="7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1"
Aug 04 18:38:19.288951 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.288899    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1"} err="failed to get container status \"7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1\": rpc error: code = NotFound desc = could not find container \"7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1\": container with ID starting with 7a3cf6da7cef7f7abbfe056c2a5fecc48bc0177d47ffd2720b4e59e700ec7cb1 not found: ID does not exist"
Aug 04 18:38:19.301138 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.301115    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"]
Aug 04 18:38:19.306483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:19.306461    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25"]
Aug 04 18:38:20.111463 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:20.111417    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-9lh25" podUID="d3440818-4ee0-477f-b822-dad5fdb33d8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Aug 04 18:38:20.544636 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:38:20.544587    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3440818-4ee0-477f-b822-dad5fdb33d8a" path="/var/lib/kubelet/pods/d3440818-4ee0-477f-b822-dad5fdb33d8a/volumes"
Aug 04 18:39:43.323580 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.323541    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"]
Aug 04 18:39:43.324121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.323926    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3440818-4ee0-477f-b822-dad5fdb33d8a" containerName="main"
Aug 04 18:39:43.324121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.323944    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3440818-4ee0-477f-b822-dad5fdb33d8a" containerName="main"
Aug 04 18:39:43.324121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.323963    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f5c779-4ea7-4122-a523-ce8aae3f93a0" containerName="main"
Aug 04 18:39:43.324121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.323971    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f5c779-4ea7-4122-a523-ce8aae3f93a0" containerName="main"
Aug 04 18:39:43.324121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.324049    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3440818-4ee0-477f-b822-dad5fdb33d8a" containerName="main"
Aug 04 18:39:43.324121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.324061    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="29f5c779-4ea7-4122-a523-ce8aae3f93a0" containerName="main"
Aug 04 18:39:43.326680 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.326659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.330565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.330541    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Aug 04 18:39:43.330720 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.330578    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Aug 04 18:39:43.330720 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.330667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Aug 04 18:39:43.330720 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.330701    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-nr6j4\""
Aug 04 18:39:43.336092 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.336068    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"]
Aug 04 18:39:43.464645 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.464589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc585290-c572-480c-8899-e542c20e2388-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.464645 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.464652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.464872 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.464717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.464872 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.464732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-home\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.464872 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.464748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-dshm\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.464872 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.464764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bf8\" (UniqueName: \"kubernetes.io/projected/fc585290-c572-480c-8899-e542c20e2388-kube-api-access-g9bf8\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.565706 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.565666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.565706 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.565706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-home\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.565979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.565723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-dshm\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.565979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.565740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bf8\" (UniqueName: \"kubernetes.io/projected/fc585290-c572-480c-8899-e542c20e2388-kube-api-access-g9bf8\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.565979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.565767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc585290-c572-480c-8899-e542c20e2388-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.565979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.565887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.566195 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.566178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.566250 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.566230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-home\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.566287 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.566259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.568044 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.568014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-dshm\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.568408 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.568388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc585290-c572-480c-8899-e542c20e2388-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.575294 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.575237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bf8\" (UniqueName: \"kubernetes.io/projected/fc585290-c572-480c-8899-e542c20e2388-kube-api-access-g9bf8\") pod \"roll-v1-kserve-69cf8bcfc-n48qf\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.637326 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.637289    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:43.773968 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:43.773936    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"]
Aug 04 18:39:43.776962 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:39:43.776930    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc585290_c572_480c_8899_e542c20e2388.slice/crio-a01a68f1a893db9cd5525b227c10e837d9a669bd85bb18f875a39bcd0b682eac WatchSource:0}: Error finding container a01a68f1a893db9cd5525b227c10e837d9a669bd85bb18f875a39bcd0b682eac: Status 404 returned error can't find the container with id a01a68f1a893db9cd5525b227c10e837d9a669bd85bb18f875a39bcd0b682eac
Aug 04 18:39:44.599476 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:44.599438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf" event={"ID":"fc585290-c572-480c-8899-e542c20e2388","Type":"ContainerStarted","Data":"414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45"}
Aug 04 18:39:44.599864 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:44.599491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf" event={"ID":"fc585290-c572-480c-8899-e542c20e2388","Type":"ContainerStarted","Data":"a01a68f1a893db9cd5525b227c10e837d9a669bd85bb18f875a39bcd0b682eac"}
Aug 04 18:39:44.618342 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:44.618291    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf" podStartSLOduration=1.618276486 podStartE2EDuration="1.618276486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:39:44.617090431 +0000 UTC m=+2392.551160330" watchObservedRunningTime="2026-08-04 18:39:44.618276486 +0000 UTC m=+2392.552346385"
Aug 04 18:39:44.638125 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:44.638097    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:53.638249 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:53.638213    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:53.650817 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:53.650792    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:39:54.648531 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:39:54.648488    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:41:04.131200 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:04.131162    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"]
Aug 04 18:41:19.149806 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.149747    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf" podUID="fc585290-c572-480c-8899-e542c20e2388" containerName="main" containerID="cri-o://414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45" gracePeriod=45
Aug 04 18:41:19.392200 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.392171    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:41:19.439237 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439161    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-tmp-dir\") pod \"fc585290-c572-480c-8899-e542c20e2388\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") "
Aug 04 18:41:19.439237 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439218    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-model-cache\") pod \"fc585290-c572-480c-8899-e542c20e2388\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") "
Aug 04 18:41:19.439449 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-dshm\") pod \"fc585290-c572-480c-8899-e542c20e2388\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") "
Aug 04 18:41:19.439449 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-home\") pod \"fc585290-c572-480c-8899-e542c20e2388\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") "
Aug 04 18:41:19.439449 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9bf8\" (UniqueName: \"kubernetes.io/projected/fc585290-c572-480c-8899-e542c20e2388-kube-api-access-g9bf8\") pod \"fc585290-c572-480c-8899-e542c20e2388\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") "
Aug 04 18:41:19.439449 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc585290-c572-480c-8899-e542c20e2388-tls-certs\") pod \"fc585290-c572-480c-8899-e542c20e2388\" (UID: \"fc585290-c572-480c-8899-e542c20e2388\") "
Aug 04 18:41:19.439689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc585290-c572-480c-8899-e542c20e2388" (UID: "fc585290-c572-480c-8899-e542c20e2388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:41:19.439689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439545    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc585290-c572-480c-8899-e542c20e2388" (UID: "fc585290-c572-480c-8899-e542c20e2388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:41:19.439689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439558    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-home" (OuterVolumeSpecName: "home") pod "fc585290-c572-480c-8899-e542c20e2388" (UID: "fc585290-c572-480c-8899-e542c20e2388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:41:19.439689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.439582    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.441574 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.441550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-dshm" (OuterVolumeSpecName: "dshm") pod "fc585290-c572-480c-8899-e542c20e2388" (UID: "fc585290-c572-480c-8899-e542c20e2388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:41:19.441668 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.441623    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc585290-c572-480c-8899-e542c20e2388-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc585290-c572-480c-8899-e542c20e2388" (UID: "fc585290-c572-480c-8899-e542c20e2388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:41:19.441707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.441685    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc585290-c572-480c-8899-e542c20e2388-kube-api-access-g9bf8" (OuterVolumeSpecName: "kube-api-access-g9bf8") pod "fc585290-c572-480c-8899-e542c20e2388" (UID: "fc585290-c572-480c-8899-e542c20e2388"). InnerVolumeSpecName "kube-api-access-g9bf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:41:19.540855 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.540815    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.540855 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.540846    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.540855 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.540854    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc585290-c572-480c-8899-e542c20e2388-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.540855 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.540863    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9bf8\" (UniqueName: \"kubernetes.io/projected/fc585290-c572-480c-8899-e542c20e2388-kube-api-access-g9bf8\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.541124 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.540872    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc585290-c572-480c-8899-e542c20e2388-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.958708 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.958666    2581 generic.go:356] "Generic (PLEG): container finished" podID="fc585290-c572-480c-8899-e542c20e2388" containerID="414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45" exitCode=0
Aug 04 18:41:19.958878 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.958721    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"
Aug 04 18:41:19.958878 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.958735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf" event={"ID":"fc585290-c572-480c-8899-e542c20e2388","Type":"ContainerDied","Data":"414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45"}
Aug 04 18:41:19.958878 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.958789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf" event={"ID":"fc585290-c572-480c-8899-e542c20e2388","Type":"ContainerDied","Data":"a01a68f1a893db9cd5525b227c10e837d9a669bd85bb18f875a39bcd0b682eac"}
Aug 04 18:41:19.958878 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.958818    2581 scope.go:117] "RemoveContainer" containerID="414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45"
Aug 04 18:41:19.967236 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.967213    2581 scope.go:117] "RemoveContainer" containerID="414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45"
Aug 04 18:41:19.967487 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:41:19.967466    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45\": container with ID starting with 414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45 not found: ID does not exist" containerID="414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45"
Aug 04 18:41:19.967540 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.967496    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45"} err="failed to get container status \"414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45\": rpc error: code = NotFound desc = could not find container \"414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45\": container with ID starting with 414f4ddc483a3b8ec1f4ea63dda937e54ecd92e0033098fb89357f73f86eaa45 not found: ID does not exist"
Aug 04 18:41:19.980794 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.980773    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"]
Aug 04 18:41:19.986612 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:19.986580    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-n48qf"]
Aug 04 18:41:20.545137 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:20.545095    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc585290-c572-480c-8899-e542c20e2388" path="/var/lib/kubelet/pods/fc585290-c572-480c-8899-e542c20e2388/volumes"
Aug 04 18:41:25.105697 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.105653    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"]
Aug 04 18:41:25.106279 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.106261    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc585290-c572-480c-8899-e542c20e2388" containerName="main"
Aug 04 18:41:25.106370 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.106282    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc585290-c572-480c-8899-e542c20e2388" containerName="main"
Aug 04 18:41:25.106454 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.106440    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc585290-c572-480c-8899-e542c20e2388" containerName="main"
Aug 04 18:41:25.111395 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.111371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.114084 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.114066    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Aug 04 18:41:25.115534 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.115517    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Aug 04 18:41:25.115534 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.115541    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-pmzcb\""
Aug 04 18:41:25.115710 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.115618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Aug 04 18:41:25.119040 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.119020    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"]
Aug 04 18:41:25.181367 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.181332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89v6\" (UniqueName: \"kubernetes.io/projected/0bd3bbf5-8526-45d3-934a-ddec97d4f715-kube-api-access-v89v6\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.181565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.181391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-dshm\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.181565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.181423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-home\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.181565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.181468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.181565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.181499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-model-cache\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.181565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.181523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.282086 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.282048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89v6\" (UniqueName: \"kubernetes.io/projected/0bd3bbf5-8526-45d3-934a-ddec97d4f715-kube-api-access-v89v6\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.282280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.282100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-dshm\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.282280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.282128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-home\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.282280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.282156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.282280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.282182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-model-cache\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.282280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.282202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.282561 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.282539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-home\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.282697 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.282671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.282697 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.282719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-model-cache\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.284510 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.284490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-dshm\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.284682 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.284665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.291201 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.291178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89v6\" (UniqueName: \"kubernetes.io/projected/0bd3bbf5-8526-45d3-934a-ddec97d4f715-kube-api-access-v89v6\") pod \"owner-v1-kserve-75579ff5bc-sdt47\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.421587 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.421558    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:25.554322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.554295    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"]
Aug 04 18:41:25.556291 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:41:25.556264    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd3bbf5_8526_45d3_934a_ddec97d4f715.slice/crio-9509051e91ed039cc1f959bc3fb6965aaf2c7bd07c6b7825ccad70d86ea13966 WatchSource:0}: Error finding container 9509051e91ed039cc1f959bc3fb6965aaf2c7bd07c6b7825ccad70d86ea13966: Status 404 returned error can't find the container with id 9509051e91ed039cc1f959bc3fb6965aaf2c7bd07c6b7825ccad70d86ea13966
Aug 04 18:41:25.983482 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.983448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47" event={"ID":"0bd3bbf5-8526-45d3-934a-ddec97d4f715","Type":"ContainerStarted","Data":"c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7"}
Aug 04 18:41:25.983644 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:25.983499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47" event={"ID":"0bd3bbf5-8526-45d3-934a-ddec97d4f715","Type":"ContainerStarted","Data":"9509051e91ed039cc1f959bc3fb6965aaf2c7bd07c6b7825ccad70d86ea13966"}
Aug 04 18:41:26.002549 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:26.002492    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47" podStartSLOduration=1.002470053 podStartE2EDuration="1.002470053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:26.001147454 +0000 UTC m=+2493.935217380" watchObservedRunningTime="2026-08-04 18:41:26.002470053 +0000 UTC m=+2493.936539954"
Aug 04 18:41:26.422532 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:26.422495    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:35.422700 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:35.422661    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:35.434968 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:35.434938    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:41:36.035422 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:41:36.035389    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:42:35.274641 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:35.274598    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"]
Aug 04 18:42:50.293706 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.293633    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47" podUID="0bd3bbf5-8526-45d3-934a-ddec97d4f715" containerName="main" containerID="cri-o://c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7" gracePeriod=45
Aug 04 18:42:50.526630 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.526588    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:42:50.691828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.691777    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-dshm\") pod \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") "
Aug 04 18:42:50.691828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.691834    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-model-cache\") pod \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") "
Aug 04 18:42:50.692090 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.691861    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v89v6\" (UniqueName: \"kubernetes.io/projected/0bd3bbf5-8526-45d3-934a-ddec97d4f715-kube-api-access-v89v6\") pod \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") "
Aug 04 18:42:50.692090 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.691900    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tls-certs\") pod \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") "
Aug 04 18:42:50.692090 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.691951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-home\") pod \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") "
Aug 04 18:42:50.692090 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.691971    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tmp-dir\") pod \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\" (UID: \"0bd3bbf5-8526-45d3-934a-ddec97d4f715\") "
Aug 04 18:42:50.692306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.692160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bd3bbf5-8526-45d3-934a-ddec97d4f715" (UID: "0bd3bbf5-8526-45d3-934a-ddec97d4f715"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:50.692306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.692247    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-home" (OuterVolumeSpecName: "home") pod "0bd3bbf5-8526-45d3-934a-ddec97d4f715" (UID: "0bd3bbf5-8526-45d3-934a-ddec97d4f715"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:50.692410 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.692326    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bd3bbf5-8526-45d3-934a-ddec97d4f715" (UID: "0bd3bbf5-8526-45d3-934a-ddec97d4f715"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:50.694184 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.694152    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bd3bbf5-8526-45d3-934a-ddec97d4f715" (UID: "0bd3bbf5-8526-45d3-934a-ddec97d4f715"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:42:50.694288 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.694190    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-dshm" (OuterVolumeSpecName: "dshm") pod "0bd3bbf5-8526-45d3-934a-ddec97d4f715" (UID: "0bd3bbf5-8526-45d3-934a-ddec97d4f715"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:50.694288 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.694217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd3bbf5-8526-45d3-934a-ddec97d4f715-kube-api-access-v89v6" (OuterVolumeSpecName: "kube-api-access-v89v6") pod "0bd3bbf5-8526-45d3-934a-ddec97d4f715" (UID: "0bd3bbf5-8526-45d3-934a-ddec97d4f715"). InnerVolumeSpecName "kube-api-access-v89v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:42:50.792825 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.792767    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:42:50.792825 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.792835    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:42:50.792825 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.792847    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:42:50.793073 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.792855    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3bbf5-8526-45d3-934a-ddec97d4f715-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:42:50.793073 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.792865    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89v6\" (UniqueName: \"kubernetes.io/projected/0bd3bbf5-8526-45d3-934a-ddec97d4f715-kube-api-access-v89v6\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:42:50.793073 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:50.792874    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3bbf5-8526-45d3-934a-ddec97d4f715-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:42:51.332785 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:51.332746    2581 generic.go:356] "Generic (PLEG): container finished" podID="0bd3bbf5-8526-45d3-934a-ddec97d4f715" containerID="c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7" exitCode=0
Aug 04 18:42:51.332785 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:51.332780    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"
Aug 04 18:42:51.333376 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:51.332808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47" event={"ID":"0bd3bbf5-8526-45d3-934a-ddec97d4f715","Type":"ContainerDied","Data":"c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7"}
Aug 04 18:42:51.333376 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:51.332862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47" event={"ID":"0bd3bbf5-8526-45d3-934a-ddec97d4f715","Type":"ContainerDied","Data":"9509051e91ed039cc1f959bc3fb6965aaf2c7bd07c6b7825ccad70d86ea13966"}
Aug 04 18:42:51.333376 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:51.332886    2581 scope.go:117] "RemoveContainer" containerID="c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7"
Aug 04 18:42:51.341403 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:51.341388    2581 scope.go:117] "RemoveContainer" containerID="c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7"
Aug 04 18:42:51.341689 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:42:51.341667    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7\": container with ID starting with c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7 not found: ID does not exist" containerID="c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7"
Aug 04 18:42:51.341747 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:51.341699    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7"} err="failed to get container status \"c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7\": rpc error: code = NotFound desc = could not find container \"c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7\": container with ID starting with c6bc011c8b4d4619996e8f311624819cd9cafebeca6e102af5e4a6a0cb85b8a7 not found: ID does not exist"
Aug 04 18:42:51.356428 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:51.356406    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"]
Aug 04 18:42:51.362579 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:51.362556    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-sdt47"]
Aug 04 18:42:52.544866 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:42:52.544833    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd3bbf5-8526-45d3-934a-ddec97d4f715" path="/var/lib/kubelet/pods/0bd3bbf5-8526-45d3-934a-ddec97d4f715/volumes"
Aug 04 18:43:28.499766 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.499692    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"]
Aug 04 18:43:28.500228 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.499986    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd3bbf5-8526-45d3-934a-ddec97d4f715" containerName="main"
Aug 04 18:43:28.500228 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.499997    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd3bbf5-8526-45d3-934a-ddec97d4f715" containerName="main"
Aug 04 18:43:28.500228 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.500048    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd3bbf5-8526-45d3-934a-ddec97d4f715" containerName="main"
Aug 04 18:43:28.502879 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.502863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.505889 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.505870    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Aug 04 18:43:28.507495 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.507473    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Aug 04 18:43:28.507631 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.507493    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Aug 04 18:43:28.507631 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.507497    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-vw2hk\""
Aug 04 18:43:28.515488 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.515468    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"]
Aug 04 18:43:28.582743 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.582712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.582743 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.582744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574d45a0-2b17-403a-a263-94eaea35478b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.582938 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.582765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.582938 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.582845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.582938 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.582894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.582938 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.582926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nprm\" (UniqueName: \"kubernetes.io/projected/574d45a0-2b17-403a-a263-94eaea35478b-kube-api-access-2nprm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.683905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.683875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.683905 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.683910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574d45a0-2b17-403a-a263-94eaea35478b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.684109 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.683935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.684109 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.683977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.684109 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.684004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.684262 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.684178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nprm\" (UniqueName: \"kubernetes.io/projected/574d45a0-2b17-403a-a263-94eaea35478b-kube-api-access-2nprm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.684392 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.684366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.684392 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.684387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.684537 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.684465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.686243 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.686223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.686411 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.686393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574d45a0-2b17-403a-a263-94eaea35478b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.692792 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.692771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nprm\" (UniqueName: \"kubernetes.io/projected/574d45a0-2b17-403a-a263-94eaea35478b-kube-api-access-2nprm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.812658 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.812555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:28.940585 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:28.940552    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"]
Aug 04 18:43:28.942882 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:43:28.942852    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574d45a0_2b17_403a_a263_94eaea35478b.slice/crio-52823a6ff5655e6240ccac8ff9e0ec9937049b7e3c91bea8c9504a7734187961 WatchSource:0}: Error finding container 52823a6ff5655e6240ccac8ff9e0ec9937049b7e3c91bea8c9504a7734187961: Status 404 returned error can't find the container with id 52823a6ff5655e6240ccac8ff9e0ec9937049b7e3c91bea8c9504a7734187961
Aug 04 18:43:29.476115 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:29.476081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v" event={"ID":"574d45a0-2b17-403a-a263-94eaea35478b","Type":"ContainerStarted","Data":"947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b"}
Aug 04 18:43:29.476289 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:29.476144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v" event={"ID":"574d45a0-2b17-403a-a263-94eaea35478b","Type":"ContainerStarted","Data":"52823a6ff5655e6240ccac8ff9e0ec9937049b7e3c91bea8c9504a7734187961"}
Aug 04 18:43:29.496659 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:29.496615    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v" podStartSLOduration=1.496591241 podStartE2EDuration="1.496591241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:43:29.494362234 +0000 UTC m=+2617.428432139" watchObservedRunningTime="2026-08-04 18:43:29.496591241 +0000 UTC m=+2617.430661139"
Aug 04 18:43:29.813816 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:29.813727    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:38.813777 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:38.813731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:38.826435 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:38.826406    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:39.523619 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:39.523571    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:43:48.297033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.297003    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"]
Aug 04 18:43:48.299321 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.299302    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:48.302336 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.302315    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Aug 04 18:43:48.302451 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.302323    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Aug 04 18:43:48.303801 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.303784    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-dqwtt\""
Aug 04 18:43:48.303889 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.303821    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Aug 04 18:43:48.309321 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.309303    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"]
Aug 04 18:43:48.445041 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.445006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364061ad-5a9d-47c4-b677-b8e9c504041f-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7\" (UID: \"364061ad-5a9d-47c4-b677-b8e9c504041f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:48.445216 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.445048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnv9w\" (UniqueName: \"kubernetes.io/projected/364061ad-5a9d-47c4-b677-b8e9c504041f-kube-api-access-jnv9w\") pod \"e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7\" (UID: \"364061ad-5a9d-47c4-b677-b8e9c504041f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:48.545596 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.545566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364061ad-5a9d-47c4-b677-b8e9c504041f-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7\" (UID: \"364061ad-5a9d-47c4-b677-b8e9c504041f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:48.545811 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.545616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnv9w\" (UniqueName: \"kubernetes.io/projected/364061ad-5a9d-47c4-b677-b8e9c504041f-kube-api-access-jnv9w\") pod \"e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7\" (UID: \"364061ad-5a9d-47c4-b677-b8e9c504041f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:48.545949 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.545913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364061ad-5a9d-47c4-b677-b8e9c504041f-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7\" (UID: \"364061ad-5a9d-47c4-b677-b8e9c504041f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:48.558985 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.558925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnv9w\" (UniqueName: \"kubernetes.io/projected/364061ad-5a9d-47c4-b677-b8e9c504041f-kube-api-access-jnv9w\") pod \"e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7\" (UID: \"364061ad-5a9d-47c4-b677-b8e9c504041f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:48.610538 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.610498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:48.738036 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:48.738012    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"]
Aug 04 18:43:48.739961 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:43:48.739934    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364061ad_5a9d_47c4_b677_b8e9c504041f.slice/crio-d6f3d8c6c3a92ab91beeaf37622f24c15115da03e1d57167cfbf340f74fd096c WatchSource:0}: Error finding container d6f3d8c6c3a92ab91beeaf37622f24c15115da03e1d57167cfbf340f74fd096c: Status 404 returned error can't find the container with id d6f3d8c6c3a92ab91beeaf37622f24c15115da03e1d57167cfbf340f74fd096c
Aug 04 18:43:49.548252 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:49.548211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7" event={"ID":"364061ad-5a9d-47c4-b677-b8e9c504041f","Type":"ContainerStarted","Data":"92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781"}
Aug 04 18:43:49.548648 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:49.548279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7" event={"ID":"364061ad-5a9d-47c4-b677-b8e9c504041f","Type":"ContainerStarted","Data":"d6f3d8c6c3a92ab91beeaf37622f24c15115da03e1d57167cfbf340f74fd096c"}
Aug 04 18:43:50.629016 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:50.628978    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"]
Aug 04 18:43:51.556091 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:51.556052    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7" podUID="364061ad-5a9d-47c4-b677-b8e9c504041f" containerName="storage-initializer" containerID="cri-o://92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781" gracePeriod=30
Aug 04 18:43:52.412027 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.412000    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:52.478687 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.478656    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364061ad-5a9d-47c4-b677-b8e9c504041f-kserve-provision-location\") pod \"364061ad-5a9d-47c4-b677-b8e9c504041f\" (UID: \"364061ad-5a9d-47c4-b677-b8e9c504041f\") "
Aug 04 18:43:52.478843 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.478698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnv9w\" (UniqueName: \"kubernetes.io/projected/364061ad-5a9d-47c4-b677-b8e9c504041f-kube-api-access-jnv9w\") pod \"364061ad-5a9d-47c4-b677-b8e9c504041f\" (UID: \"364061ad-5a9d-47c4-b677-b8e9c504041f\") "
Aug 04 18:43:52.480851 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.480824    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364061ad-5a9d-47c4-b677-b8e9c504041f-kube-api-access-jnv9w" (OuterVolumeSpecName: "kube-api-access-jnv9w") pod "364061ad-5a9d-47c4-b677-b8e9c504041f" (UID: "364061ad-5a9d-47c4-b677-b8e9c504041f"). InnerVolumeSpecName "kube-api-access-jnv9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:43:52.545370 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.545297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364061ad-5a9d-47c4-b677-b8e9c504041f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "364061ad-5a9d-47c4-b677-b8e9c504041f" (UID: "364061ad-5a9d-47c4-b677-b8e9c504041f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:43:52.560087 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.560063    2581 generic.go:356] "Generic (PLEG): container finished" podID="364061ad-5a9d-47c4-b677-b8e9c504041f" containerID="92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781" exitCode=0
Aug 04 18:43:52.560202 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.560105    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"
Aug 04 18:43:52.560202 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.560148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7" event={"ID":"364061ad-5a9d-47c4-b677-b8e9c504041f","Type":"ContainerDied","Data":"92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781"}
Aug 04 18:43:52.560202 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.560198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7" event={"ID":"364061ad-5a9d-47c4-b677-b8e9c504041f","Type":"ContainerDied","Data":"d6f3d8c6c3a92ab91beeaf37622f24c15115da03e1d57167cfbf340f74fd096c"}
Aug 04 18:43:52.560313 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.560221    2581 scope.go:117] "RemoveContainer" containerID="92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781"
Aug 04 18:43:52.568431 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.568408    2581 scope.go:117] "RemoveContainer" containerID="92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781"
Aug 04 18:43:52.568696 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:43:52.568675    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781\": container with ID starting with 92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781 not found: ID does not exist" containerID="92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781"
Aug 04 18:43:52.568773 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.568702    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781"} err="failed to get container status \"92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781\": rpc error: code = NotFound desc = could not find container \"92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781\": container with ID starting with 92f2c78e6fe2a5a36fe52cd141d92ada8a3cb5661ad0a2ffbc4cd9543d769781 not found: ID does not exist"
Aug 04 18:43:52.579756 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.579739    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364061ad-5a9d-47c4-b677-b8e9c504041f-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:43:52.579812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.579758    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnv9w\" (UniqueName: \"kubernetes.io/projected/364061ad-5a9d-47c4-b677-b8e9c504041f-kube-api-access-jnv9w\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:43:52.596033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.596010    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"]
Aug 04 18:43:52.598362 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:52.598340    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-dc5dc878d-rrsq7"]
Aug 04 18:43:54.545713 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:43:54.545678    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="364061ad-5a9d-47c4-b677-b8e9c504041f" path="/var/lib/kubelet/pods/364061ad-5a9d-47c4-b677-b8e9c504041f/volumes"
Aug 04 18:44:10.997565 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:10.997532    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"]
Aug 04 18:44:10.998033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:10.997862    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364061ad-5a9d-47c4-b677-b8e9c504041f" containerName="storage-initializer"
Aug 04 18:44:10.998033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:10.997873    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="364061ad-5a9d-47c4-b677-b8e9c504041f" containerName="storage-initializer"
Aug 04 18:44:10.998033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:10.997930    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="364061ad-5a9d-47c4-b677-b8e9c504041f" containerName="storage-initializer"
Aug 04 18:44:11.000812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.000796    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.004306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.003891    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Aug 04 18:44:11.004306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.003905    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:44:11.004306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.003927    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Aug 04 18:44:11.004306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.003949    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-9b65r\""
Aug 04 18:44:11.004306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.003897    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Aug 04 18:44:11.004306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.004237    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Aug 04 18:44:11.015193 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.015169    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"]
Aug 04 18:44:11.128226 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.128196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tls-certs\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.128400 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.128235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-model-cache\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.128400 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.128293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-home\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.128400 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.128312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kserve-provision-location\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.128400 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.128342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tmp-dir\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.128400 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.128359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jnfh\" (UniqueName: \"kubernetes.io/projected/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kube-api-access-8jnfh\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.128400 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.128374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-cabundle-cert\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.128400 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.128389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-dshm\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.229888 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.229849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-home\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.229888 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.229887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kserve-provision-location\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.229907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tmp-dir\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.229930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jnfh\" (UniqueName: \"kubernetes.io/projected/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kube-api-access-8jnfh\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.230038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-cabundle-cert\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.230076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-dshm\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230293 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.230141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tls-certs\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230293 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.230192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-model-cache\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230397 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.230292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-home\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230397 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.230372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kserve-provision-location\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230500 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.230451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tmp-dir\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230554 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.230522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-model-cache\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.230763 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.230744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-cabundle-cert\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.232456 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.232432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-dshm\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.232811 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.232791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tls-certs\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.238670 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.238648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jnfh\" (UniqueName: \"kubernetes.io/projected/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kube-api-access-8jnfh\") pod \"stop-feature-test-kserve-896f8bf9c-gdbz6\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.311282 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.311202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:11.444249 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.444147    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"]
Aug 04 18:44:11.446529 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:44:11.446500    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce54f5b_46c8_4e11_9dd8_ffa7726fba02.slice/crio-32410e2c68105c5b4b52df418b8143642c0882619453eed9f21ac30bc1baf4cd WatchSource:0}: Error finding container 32410e2c68105c5b4b52df418b8143642c0882619453eed9f21ac30bc1baf4cd: Status 404 returned error can't find the container with id 32410e2c68105c5b4b52df418b8143642c0882619453eed9f21ac30bc1baf4cd
Aug 04 18:44:11.625195 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.625107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" event={"ID":"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02","Type":"ContainerStarted","Data":"8a6a419a9726e8537c6030b423b310cc7147264e0fe195b8144804a4ab30486d"}
Aug 04 18:44:11.625195 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:11.625162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" event={"ID":"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02","Type":"ContainerStarted","Data":"32410e2c68105c5b4b52df418b8143642c0882619453eed9f21ac30bc1baf4cd"}
Aug 04 18:44:13.634214 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:13.634176    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerID="8a6a419a9726e8537c6030b423b310cc7147264e0fe195b8144804a4ab30486d" exitCode=0
Aug 04 18:44:13.634613 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:13.634247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" event={"ID":"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02","Type":"ContainerDied","Data":"8a6a419a9726e8537c6030b423b310cc7147264e0fe195b8144804a4ab30486d"}
Aug 04 18:44:14.641052 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:14.641019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" event={"ID":"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02","Type":"ContainerStarted","Data":"81e0bc8a88d409a53a001d4fefbb330d00094a0b1dd98fb55f445b15c827ddce"}
Aug 04 18:44:14.666477 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:14.666427    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podStartSLOduration=2.668532951 podStartE2EDuration="4.666410846s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.997877895s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:44:14.66490978 +0000 UTC m=+2662.598979680" watchObservedRunningTime="2026-08-04 18:44:14.666410846 +0000 UTC m=+2662.600480744"
Aug 04 18:44:15.311959 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:15.311924    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:21.311527 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:21.311488    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:44:21.312959 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:21.312932    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:44:31.312167 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:31.312111    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:44:41.312088 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:41.312040    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:44:51.311928 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:44:51.311815    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:45:01.311976 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:01.311935    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:45:11.312632 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:11.312578    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:45:21.312061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:21.312013    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:45:31.312675 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:31.312622    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:45:41.312079 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:41.312035    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:45:44.640659 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:44.640623    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"]
Aug 04 18:45:44.649260 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:45:44.649223    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:45:44.649260 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:45:44.649260 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:45:44.649260 ip-10-0-139-201 kubenswrapper[2581]:  > containerID="947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b" cmd=["/bin/sleep","15"]
Aug 04 18:45:44.649584 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:44.649280    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v" message="" err=<
Aug 04 18:45:44.649584 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:45:44.649584 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:45:44.649584 ip-10-0-139-201 kubenswrapper[2581]:  >
Aug 04 18:45:44.649584 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:45:44.649292    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:45:44.649584 ip-10-0-139-201 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:45:44.649584 ip-10-0-139-201 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Aug 04 18:45:44.649584 ip-10-0-139-201 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v" podUID="574d45a0-2b17-403a-a263-94eaea35478b" containerName="main" containerID="cri-o://947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b"
Aug 04 18:45:44.649584 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:44.649343    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v" podUID="574d45a0-2b17-403a-a263-94eaea35478b" containerName="main" containerID="cri-o://947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b" gracePeriod=60
Aug 04 18:45:44.995282 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:44.995255    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:45:45.004398 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.004368    2581 generic.go:356] "Generic (PLEG): container finished" podID="574d45a0-2b17-403a-a263-94eaea35478b" containerID="947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b" exitCode=0
Aug 04 18:45:45.004539 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.004408    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"
Aug 04 18:45:45.004539 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.004446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v" event={"ID":"574d45a0-2b17-403a-a263-94eaea35478b","Type":"ContainerDied","Data":"947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b"}
Aug 04 18:45:45.004539 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.004498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v" event={"ID":"574d45a0-2b17-403a-a263-94eaea35478b","Type":"ContainerDied","Data":"52823a6ff5655e6240ccac8ff9e0ec9937049b7e3c91bea8c9504a7734187961"}
Aug 04 18:45:45.004539 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.004522    2581 scope.go:117] "RemoveContainer" containerID="947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b"
Aug 04 18:45:45.015174 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.015155    2581 scope.go:117] "RemoveContainer" containerID="947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b"
Aug 04 18:45:45.015479 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:45:45.015459    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b\": container with ID starting with 947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b not found: ID does not exist" containerID="947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b"
Aug 04 18:45:45.015563 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.015491    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b"} err="failed to get container status \"947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b\": rpc error: code = NotFound desc = could not find container \"947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b\": container with ID starting with 947c3b4aefbbef1159f5846204068bc3cc8dc43973e0a723880992ca11ae4d1b not found: ID does not exist"
Aug 04 18:45:45.102677 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.102644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-home\") pod \"574d45a0-2b17-403a-a263-94eaea35478b\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") "
Aug 04 18:45:45.102845 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.102692    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-model-cache\") pod \"574d45a0-2b17-403a-a263-94eaea35478b\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") "
Aug 04 18:45:45.102845 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.102721    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574d45a0-2b17-403a-a263-94eaea35478b-tls-certs\") pod \"574d45a0-2b17-403a-a263-94eaea35478b\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") "
Aug 04 18:45:45.102845 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.102756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-tmp-dir\") pod \"574d45a0-2b17-403a-a263-94eaea35478b\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") "
Aug 04 18:45:45.102845 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.102800    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nprm\" (UniqueName: \"kubernetes.io/projected/574d45a0-2b17-403a-a263-94eaea35478b-kube-api-access-2nprm\") pod \"574d45a0-2b17-403a-a263-94eaea35478b\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") "
Aug 04 18:45:45.102845 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.102836    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-dshm\") pod \"574d45a0-2b17-403a-a263-94eaea35478b\" (UID: \"574d45a0-2b17-403a-a263-94eaea35478b\") "
Aug 04 18:45:45.103106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.102917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-home" (OuterVolumeSpecName: "home") pod "574d45a0-2b17-403a-a263-94eaea35478b" (UID: "574d45a0-2b17-403a-a263-94eaea35478b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:45.103106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.102939    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-model-cache" (OuterVolumeSpecName: "model-cache") pod "574d45a0-2b17-403a-a263-94eaea35478b" (UID: "574d45a0-2b17-403a-a263-94eaea35478b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:45.103106 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.103002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "574d45a0-2b17-403a-a263-94eaea35478b" (UID: "574d45a0-2b17-403a-a263-94eaea35478b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:45.103268 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.103134    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:45:45.103268 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.103153    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:45:45.103268 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.103169    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:45:45.105010 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.104981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-dshm" (OuterVolumeSpecName: "dshm") pod "574d45a0-2b17-403a-a263-94eaea35478b" (UID: "574d45a0-2b17-403a-a263-94eaea35478b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:45.105138 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.105034    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/574d45a0-2b17-403a-a263-94eaea35478b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "574d45a0-2b17-403a-a263-94eaea35478b" (UID: "574d45a0-2b17-403a-a263-94eaea35478b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:45:45.105138 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.105112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574d45a0-2b17-403a-a263-94eaea35478b-kube-api-access-2nprm" (OuterVolumeSpecName: "kube-api-access-2nprm") pod "574d45a0-2b17-403a-a263-94eaea35478b" (UID: "574d45a0-2b17-403a-a263-94eaea35478b"). InnerVolumeSpecName "kube-api-access-2nprm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:45:45.203768 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.203679    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574d45a0-2b17-403a-a263-94eaea35478b-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:45:45.203768 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.203707    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nprm\" (UniqueName: \"kubernetes.io/projected/574d45a0-2b17-403a-a263-94eaea35478b-kube-api-access-2nprm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:45:45.203768 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.203720    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574d45a0-2b17-403a-a263-94eaea35478b-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:45:45.334095 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.334060    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"]
Aug 04 18:45:45.339502 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:45.339477    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997q5d4v"]
Aug 04 18:45:46.545308 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:46.545260    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="574d45a0-2b17-403a-a263-94eaea35478b" path="/var/lib/kubelet/pods/574d45a0-2b17-403a-a263-94eaea35478b/volumes"
Aug 04 18:45:51.312293 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:45:51.312249    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:46:01.312061 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:01.312019    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Aug 04 18:46:11.322794 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:11.322762    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:46:11.330499 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:11.330470    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:46:15.478483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:15.478401    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"]
Aug 04 18:46:30.501183 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.501141    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main" containerID="cri-o://81e0bc8a88d409a53a001d4fefbb330d00094a0b1dd98fb55f445b15c827ddce" gracePeriod=45
Aug 04 18:46:30.822759 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.822680    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"]
Aug 04 18:46:30.823074 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.823058    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="574d45a0-2b17-403a-a263-94eaea35478b" containerName="main"
Aug 04 18:46:30.823116 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.823080    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="574d45a0-2b17-403a-a263-94eaea35478b" containerName="main"
Aug 04 18:46:30.823188 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.823178    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="574d45a0-2b17-403a-a263-94eaea35478b" containerName="main"
Aug 04 18:46:30.826809 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.826788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.838206 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.838180    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"]
Aug 04 18:46:30.860789 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.860759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwc9m\" (UniqueName: \"kubernetes.io/projected/8df11070-4865-49d0-b000-5339bdbae0e8-kube-api-access-gwc9m\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.860923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.860798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-tmp-dir\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.860923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.860821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-model-cache\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.860923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.860850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df11070-4865-49d0-b000-5339bdbae0e8-tls-certs\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.860923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.860877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-kserve-provision-location\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.860923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.860893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8df11070-4865-49d0-b000-5339bdbae0e8-cabundle-cert\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.860923 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.860916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-home\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.861120 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.860965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-dshm\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962152 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-tmp-dir\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962324 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-model-cache\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962324 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df11070-4865-49d0-b000-5339bdbae0e8-tls-certs\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962324 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-kserve-provision-location\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962324 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8df11070-4865-49d0-b000-5339bdbae0e8-cabundle-cert\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962651 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-home\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962651 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-dshm\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962651 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwc9m\" (UniqueName: \"kubernetes.io/projected/8df11070-4865-49d0-b000-5339bdbae0e8-kube-api-access-gwc9m\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962651 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-tmp-dir\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962651 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-model-cache\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962875 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-kserve-provision-location\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.962932 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.962913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-home\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.963139 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.963115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8df11070-4865-49d0-b000-5339bdbae0e8-cabundle-cert\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.964876 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.964853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-dshm\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.965220 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.965202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df11070-4865-49d0-b000-5339bdbae0e8-tls-certs\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:30.971417 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:30.971396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwc9m\" (UniqueName: \"kubernetes.io/projected/8df11070-4865-49d0-b000-5339bdbae0e8-kube-api-access-gwc9m\") pod \"stop-feature-test-kserve-896f8bf9c-mhcxl\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:31.137665 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.137637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:31.185400 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.185363    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerID="81e0bc8a88d409a53a001d4fefbb330d00094a0b1dd98fb55f445b15c827ddce" exitCode=0
Aug 04 18:46:31.185583 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.185412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" event={"ID":"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02","Type":"ContainerDied","Data":"81e0bc8a88d409a53a001d4fefbb330d00094a0b1dd98fb55f445b15c827ddce"}
Aug 04 18:46:31.250191 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.250161    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:46:31.264757 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.264725    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-cabundle-cert\") pod \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") "
Aug 04 18:46:31.264922 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.264773    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tmp-dir\") pod \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") "
Aug 04 18:46:31.264922 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.264796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-dshm\") pod \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") "
Aug 04 18:46:31.264922 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.264819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kserve-provision-location\") pod \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") "
Aug 04 18:46:31.264922 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.264850    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-home\") pod \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") "
Aug 04 18:46:31.264922 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.264866    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jnfh\" (UniqueName: \"kubernetes.io/projected/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kube-api-access-8jnfh\") pod \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") "
Aug 04 18:46:31.264922 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.264901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tls-certs\") pod \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") "
Aug 04 18:46:31.265251 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.264951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-model-cache\") pod \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\" (UID: \"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02\") "
Aug 04 18:46:31.265251 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.265136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" (UID: "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:46:31.265362 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.265297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" (UID: "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:46:31.265826 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.265801    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-home" (OuterVolumeSpecName: "home") pod "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" (UID: "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:46:31.267239 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.267215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" (UID: "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:46:31.267477 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.267458    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kube-api-access-8jnfh" (OuterVolumeSpecName: "kube-api-access-8jnfh") pod "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" (UID: "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02"). InnerVolumeSpecName "kube-api-access-8jnfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:46:31.267540 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.267471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-dshm" (OuterVolumeSpecName: "dshm") pod "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" (UID: "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:46:31.276549 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.276523    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" (UID: "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:46:31.288235 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.288195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" (UID: "4ce54f5b-46c8-4e11-9dd8-ffa7726fba02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:46:31.292876 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.292856    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"]
Aug 04 18:46:31.295819 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:46:31.295795    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df11070_4865_49d0_b000_5339bdbae0e8.slice/crio-4b7bdec93b63397673a2ef116c652e673a9bd75b1c99a0a7600f407b9e2aa936 WatchSource:0}: Error finding container 4b7bdec93b63397673a2ef116c652e673a9bd75b1c99a0a7600f407b9e2aa936: Status 404 returned error can't find the container with id 4b7bdec93b63397673a2ef116c652e673a9bd75b1c99a0a7600f407b9e2aa936
Aug 04 18:46:31.366131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.366076    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:46:31.366131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.366103    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:46:31.366131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.366114    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:46:31.366131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.366122    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:46:31.366131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.366132    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jnfh\" (UniqueName: \"kubernetes.io/projected/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-kube-api-access-8jnfh\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:46:31.366459 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.366140    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:46:31.366459 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.366147    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:46:31.366459 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:31.366155    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:46:32.191582 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:32.191548    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"
Aug 04 18:46:32.192068 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:32.191559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6" event={"ID":"4ce54f5b-46c8-4e11-9dd8-ffa7726fba02","Type":"ContainerDied","Data":"32410e2c68105c5b4b52df418b8143642c0882619453eed9f21ac30bc1baf4cd"}
Aug 04 18:46:32.192068 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:32.191678    2581 scope.go:117] "RemoveContainer" containerID="81e0bc8a88d409a53a001d4fefbb330d00094a0b1dd98fb55f445b15c827ddce"
Aug 04 18:46:32.193268 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:32.193239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" event={"ID":"8df11070-4865-49d0-b000-5339bdbae0e8","Type":"ContainerStarted","Data":"ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780"}
Aug 04 18:46:32.193366 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:32.193293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" event={"ID":"8df11070-4865-49d0-b000-5339bdbae0e8","Type":"ContainerStarted","Data":"4b7bdec93b63397673a2ef116c652e673a9bd75b1c99a0a7600f407b9e2aa936"}
Aug 04 18:46:32.200556 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:32.200540    2581 scope.go:117] "RemoveContainer" containerID="8a6a419a9726e8537c6030b423b310cc7147264e0fe195b8144804a4ab30486d"
Aug 04 18:46:32.230520 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:32.230467    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"]
Aug 04 18:46:32.235960 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:32.235932    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-gdbz6"]
Aug 04 18:46:32.547372 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:32.547280    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" path="/var/lib/kubelet/pods/4ce54f5b-46c8-4e11-9dd8-ffa7726fba02/volumes"
Aug 04 18:46:34.207963 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:34.207925    2581 generic.go:356] "Generic (PLEG): container finished" podID="8df11070-4865-49d0-b000-5339bdbae0e8" containerID="ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780" exitCode=0
Aug 04 18:46:34.208314 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:34.207972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" event={"ID":"8df11070-4865-49d0-b000-5339bdbae0e8","Type":"ContainerDied","Data":"ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780"}
Aug 04 18:46:35.213543 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:35.213505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" event={"ID":"8df11070-4865-49d0-b000-5339bdbae0e8","Type":"ContainerStarted","Data":"50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e"}
Aug 04 18:46:35.238200 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:35.238152    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podStartSLOduration=3.245812645 podStartE2EDuration="5.238133347s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.992320702s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:46:35.235720488 +0000 UTC m=+2803.169790389" watchObservedRunningTime="2026-08-04 18:46:35.238133347 +0000 UTC m=+2803.172203293"
Aug 04 18:46:36.138289 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:36.138243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:41.138375 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:41.138334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:46:41.140134 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:41.140106    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:46:51.138429 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:46:51.138369    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:47:01.138493 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:47:01.138440    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:47:11.138868 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:47:11.138821    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:47:21.138793 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:47:21.138751    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:47:31.138722 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:47:31.138678    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:47:41.139153 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:47:41.139103    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:47:51.138331 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:47:51.138240    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:48:01.138930 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:48:01.138875    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:48:11.138873 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:48:11.138829    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:48:21.138385 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:48:21.138340    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:48:31.148306 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:48:31.148273    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:48:31.156382 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:48:31.156355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:49:06.027827 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:06.027795    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"]
Aug 04 18:49:21.051959 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.051873    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" containerID="cri-o://50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e" gracePeriod=45
Aug 04 18:49:21.149144 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.149105    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Aug 04 18:49:21.702779 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.702756    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:49:21.775126 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775089    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-home\") pod \"8df11070-4865-49d0-b000-5339bdbae0e8\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") "
Aug 04 18:49:21.775278 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df11070-4865-49d0-b000-5339bdbae0e8-tls-certs\") pod \"8df11070-4865-49d0-b000-5339bdbae0e8\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") "
Aug 04 18:49:21.775278 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-dshm\") pod \"8df11070-4865-49d0-b000-5339bdbae0e8\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") "
Aug 04 18:49:21.775278 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775222    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-tmp-dir\") pod \"8df11070-4865-49d0-b000-5339bdbae0e8\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") "
Aug 04 18:49:21.775278 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775271    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-kserve-provision-location\") pod \"8df11070-4865-49d0-b000-5339bdbae0e8\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") "
Aug 04 18:49:21.775481 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775302    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-model-cache\") pod \"8df11070-4865-49d0-b000-5339bdbae0e8\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") "
Aug 04 18:49:21.775481 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775328    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwc9m\" (UniqueName: \"kubernetes.io/projected/8df11070-4865-49d0-b000-5339bdbae0e8-kube-api-access-gwc9m\") pod \"8df11070-4865-49d0-b000-5339bdbae0e8\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") "
Aug 04 18:49:21.775481 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775360    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8df11070-4865-49d0-b000-5339bdbae0e8-cabundle-cert\") pod \"8df11070-4865-49d0-b000-5339bdbae0e8\" (UID: \"8df11070-4865-49d0-b000-5339bdbae0e8\") "
Aug 04 18:49:21.775671 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775534    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "8df11070-4865-49d0-b000-5339bdbae0e8" (UID: "8df11070-4865-49d0-b000-5339bdbae0e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:49:21.775729 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775683    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:49:21.775935 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775898    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df11070-4865-49d0-b000-5339bdbae0e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8df11070-4865-49d0-b000-5339bdbae0e8" (UID: "8df11070-4865-49d0-b000-5339bdbae0e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:49:21.775935 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.775900    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-home" (OuterVolumeSpecName: "home") pod "8df11070-4865-49d0-b000-5339bdbae0e8" (UID: "8df11070-4865-49d0-b000-5339bdbae0e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:49:21.777484 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.777449    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-dshm" (OuterVolumeSpecName: "dshm") pod "8df11070-4865-49d0-b000-5339bdbae0e8" (UID: "8df11070-4865-49d0-b000-5339bdbae0e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:49:21.777639 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.777591    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df11070-4865-49d0-b000-5339bdbae0e8-kube-api-access-gwc9m" (OuterVolumeSpecName: "kube-api-access-gwc9m") pod "8df11070-4865-49d0-b000-5339bdbae0e8" (UID: "8df11070-4865-49d0-b000-5339bdbae0e8"). InnerVolumeSpecName "kube-api-access-gwc9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:49:21.777787 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.777766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df11070-4865-49d0-b000-5339bdbae0e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8df11070-4865-49d0-b000-5339bdbae0e8" (UID: "8df11070-4865-49d0-b000-5339bdbae0e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:49:21.793195 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.793171    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8df11070-4865-49d0-b000-5339bdbae0e8" (UID: "8df11070-4865-49d0-b000-5339bdbae0e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:49:21.799457 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.799435    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8df11070-4865-49d0-b000-5339bdbae0e8" (UID: "8df11070-4865-49d0-b000-5339bdbae0e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:49:21.839855 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.839824    2581 generic.go:356] "Generic (PLEG): container finished" podID="8df11070-4865-49d0-b000-5339bdbae0e8" containerID="50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e" exitCode=0
Aug 04 18:49:21.839991 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.839873    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"
Aug 04 18:49:21.839991 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.839888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" event={"ID":"8df11070-4865-49d0-b000-5339bdbae0e8","Type":"ContainerDied","Data":"50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e"}
Aug 04 18:49:21.839991 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.839951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl" event={"ID":"8df11070-4865-49d0-b000-5339bdbae0e8","Type":"ContainerDied","Data":"4b7bdec93b63397673a2ef116c652e673a9bd75b1c99a0a7600f407b9e2aa936"}
Aug 04 18:49:21.839991 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.839975    2581 scope.go:117] "RemoveContainer" containerID="50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e"
Aug 04 18:49:21.849709 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.849692    2581 scope.go:117] "RemoveContainer" containerID="ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780"
Aug 04 18:49:21.856757 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.856737    2581 scope.go:117] "RemoveContainer" containerID="50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e"
Aug 04 18:49:21.857004 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:49:21.856984    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e\": container with ID starting with 50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e not found: ID does not exist" containerID="50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e"
Aug 04 18:49:21.857068 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.857012    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e"} err="failed to get container status \"50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e\": rpc error: code = NotFound desc = could not find container \"50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e\": container with ID starting with 50b1293f442dd6624d88f1497a141b82955217e87722bf0990388a4b24d5aa0e not found: ID does not exist"
Aug 04 18:49:21.857068 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.857030    2581 scope.go:117] "RemoveContainer" containerID="ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780"
Aug 04 18:49:21.857266 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:49:21.857248    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780\": container with ID starting with ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780 not found: ID does not exist" containerID="ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780"
Aug 04 18:49:21.857304 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.857273    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780"} err="failed to get container status \"ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780\": rpc error: code = NotFound desc = could not find container \"ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780\": container with ID starting with ecf4c0946b40d52402b02e312915ad6527aecfc3ea4100505be37ae72a6fa780 not found: ID does not exist"
Aug 04 18:49:21.866207 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.866186    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"]
Aug 04 18:49:21.870026 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.870006    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-896f8bf9c-mhcxl"]
Aug 04 18:49:21.875961 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.875945    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:49:21.876038 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.875964    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df11070-4865-49d0-b000-5339bdbae0e8-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:49:21.876038 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.875973    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:49:21.876038 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.875980    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:49:21.876038 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.875988    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8df11070-4865-49d0-b000-5339bdbae0e8-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:49:21.876038 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.875997    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwc9m\" (UniqueName: \"kubernetes.io/projected/8df11070-4865-49d0-b000-5339bdbae0e8-kube-api-access-gwc9m\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:49:21.876038 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:21.876005    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8df11070-4865-49d0-b000-5339bdbae0e8-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:49:22.545111 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:22.545068    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" path="/var/lib/kubelet/pods/8df11070-4865-49d0-b000-5339bdbae0e8/volumes"
Aug 04 18:49:44.295167 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295132    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"]
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295390    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main"
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295400    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main"
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295411    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="storage-initializer"
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295417    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="storage-initializer"
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295426    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="storage-initializer"
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295431    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="storage-initializer"
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295438    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main"
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295443    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main"
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295490    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce54f5b-46c8-4e11-9dd8-ffa7726fba02" containerName="main"
Aug 04 18:49:44.295546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.295497    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df11070-4865-49d0-b000-5339bdbae0e8" containerName="main"
Aug 04 18:49:44.298364 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.298349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.301243 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.301221    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Aug 04 18:49:44.301423 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.301403    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Aug 04 18:49:44.301495 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.301436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Aug 04 18:49:44.301773 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.301755    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:49:44.302548 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.302531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Aug 04 18:49:44.302616 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.302530    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-k7zh7\""
Aug 04 18:49:44.314598 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.314577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"]
Aug 04 18:49:44.316805 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.316784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.316908 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.316819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-model-cache\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.316908 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.316851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tmp-dir\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.316908 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.316891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-dshm\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.317067 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.316916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6wqq\" (UniqueName: \"kubernetes.io/projected/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kube-api-access-m6wqq\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.317067 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.316954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-cabundle-cert\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.317067 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.316985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-home\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.317067 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.317017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tls-certs\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.417780 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.417742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.417780 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.417787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-model-cache\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.417818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tmp-dir\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.417846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-dshm\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.417869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6wqq\" (UniqueName: \"kubernetes.io/projected/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kube-api-access-m6wqq\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.417898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-cabundle-cert\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.417957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-home\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418033 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.417983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tls-certs\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418340 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.418219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418340 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.418260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tmp-dir\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418513 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.418340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-model-cache\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418513 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.418464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-home\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.418691 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.418673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-cabundle-cert\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.420274 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.420241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-dshm\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.420483 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.420465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tls-certs\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.427116 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.427092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6wqq\" (UniqueName: \"kubernetes.io/projected/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kube-api-access-m6wqq\") pod \"lora-single-adapter-test-kserve-785f6746f-mx77v\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.608663 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.608550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:44.738322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.738290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"]
Aug 04 18:49:44.741295 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:49:44.741267    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdca2b4a_d2a2_42ac_bfad_cad320e641fb.slice/crio-e27446473253bbb70d5ca33d0dafc64229a46da3ab6a0ec8ad1e342117b4032b WatchSource:0}: Error finding container e27446473253bbb70d5ca33d0dafc64229a46da3ab6a0ec8ad1e342117b4032b: Status 404 returned error can't find the container with id e27446473253bbb70d5ca33d0dafc64229a46da3ab6a0ec8ad1e342117b4032b
Aug 04 18:49:44.923813 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.923761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" event={"ID":"bdca2b4a-d2a2-42ac-bfad-cad320e641fb","Type":"ContainerStarted","Data":"aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4"}
Aug 04 18:49:44.924024 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:44.923848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" event={"ID":"bdca2b4a-d2a2-42ac-bfad-cad320e641fb","Type":"ContainerStarted","Data":"e27446473253bbb70d5ca33d0dafc64229a46da3ab6a0ec8ad1e342117b4032b"}
Aug 04 18:49:47.938217 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:47.938179    2581 generic.go:356] "Generic (PLEG): container finished" podID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerID="aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4" exitCode=0
Aug 04 18:49:47.938593 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:47.938255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" event={"ID":"bdca2b4a-d2a2-42ac-bfad-cad320e641fb","Type":"ContainerDied","Data":"aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4"}
Aug 04 18:49:48.944225 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:48.944183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" event={"ID":"bdca2b4a-d2a2-42ac-bfad-cad320e641fb","Type":"ContainerStarted","Data":"d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738"}
Aug 04 18:49:48.976617 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:48.976525    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podStartSLOduration=2.195981133 podStartE2EDuration="4.976502411s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.780521278s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:48.97516226 +0000 UTC m=+2996.909232160" watchObservedRunningTime="2026-08-04 18:49:48.976502411 +0000 UTC m=+2996.910572310"
Aug 04 18:49:49.608951 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:49.608905    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:54.609457 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:54.609419    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:49:54.611212 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:49:54.611176    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Aug 04 18:50:04.609281 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:50:04.609229    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Aug 04 18:50:14.609525 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:50:14.609475    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Aug 04 18:50:24.610049 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:50:24.609990    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Aug 04 18:50:34.609271 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:50:34.609219    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Aug 04 18:50:44.609004 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:50:44.608909    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Aug 04 18:50:54.609477 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:50:54.609422    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Aug 04 18:51:04.619791 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:04.619759    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:51:04.628287 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:04.628261    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:51:29.085025 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:29.084989    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-785f6746f-mx77v_bdca2b4a-d2a2-42ac-bfad-cad320e641fb/storage-initializer/0.log"
Aug 04 18:51:29.094546 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:29.094519    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-785f6746f-mx77v_bdca2b4a-d2a2-42ac-bfad-cad320e641fb/main/0.log"
Aug 04 18:51:30.504785 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:30.504751    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"]
Aug 04 18:51:45.526575 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:45.526524    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main" containerID="cri-o://d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738" gracePeriod=45
Aug 04 18:51:45.620425 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:45.620375    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Aug 04 18:51:46.173132 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.173103    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:51:46.231836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.231737    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kserve-provision-location\") pod \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") "
Aug 04 18:51:46.231836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.231778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-model-cache\") pod \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") "
Aug 04 18:51:46.231836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.231806    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-dshm\") pod \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") "
Aug 04 18:51:46.231836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.231827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-home\") pod \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") "
Aug 04 18:51:46.232186 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.231855    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tls-certs\") pod \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") "
Aug 04 18:51:46.232186 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.231885    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6wqq\" (UniqueName: \"kubernetes.io/projected/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kube-api-access-m6wqq\") pod \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") "
Aug 04 18:51:46.232186 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.231913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-cabundle-cert\") pod \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") "
Aug 04 18:51:46.232186 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.231980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tmp-dir\") pod \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\" (UID: \"bdca2b4a-d2a2-42ac-bfad-cad320e641fb\") "
Aug 04 18:51:46.232186 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.232080    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdca2b4a-d2a2-42ac-bfad-cad320e641fb" (UID: "bdca2b4a-d2a2-42ac-bfad-cad320e641fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:46.232441 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.232351    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-home" (OuterVolumeSpecName: "home") pod "bdca2b4a-d2a2-42ac-bfad-cad320e641fb" (UID: "bdca2b4a-d2a2-42ac-bfad-cad320e641fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:46.232441 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.232427    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:51:46.232549 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.232484    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bdca2b4a-d2a2-42ac-bfad-cad320e641fb" (UID: "bdca2b4a-d2a2-42ac-bfad-cad320e641fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:51:46.234224 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.234200    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-dshm" (OuterVolumeSpecName: "dshm") pod "bdca2b4a-d2a2-42ac-bfad-cad320e641fb" (UID: "bdca2b4a-d2a2-42ac-bfad-cad320e641fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:46.234368 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.234266    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kube-api-access-m6wqq" (OuterVolumeSpecName: "kube-api-access-m6wqq") pod "bdca2b4a-d2a2-42ac-bfad-cad320e641fb" (UID: "bdca2b4a-d2a2-42ac-bfad-cad320e641fb"). InnerVolumeSpecName "kube-api-access-m6wqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:51:46.234368 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.234316    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdca2b4a-d2a2-42ac-bfad-cad320e641fb" (UID: "bdca2b4a-d2a2-42ac-bfad-cad320e641fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:51:46.244404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.244376    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdca2b4a-d2a2-42ac-bfad-cad320e641fb" (UID: "bdca2b4a-d2a2-42ac-bfad-cad320e641fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:46.255079 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.255049    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdca2b4a-d2a2-42ac-bfad-cad320e641fb" (UID: "bdca2b4a-d2a2-42ac-bfad-cad320e641fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:46.333247 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.333213    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:51:46.333247 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.333246    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:51:46.333404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.333255    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:51:46.333404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.333264    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:51:46.333404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.333273    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:51:46.333404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.333281    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:51:46.333404 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.333290    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6wqq\" (UniqueName: \"kubernetes.io/projected/bdca2b4a-d2a2-42ac-bfad-cad320e641fb-kube-api-access-m6wqq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:51:46.415234 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.415203    2581 generic.go:356] "Generic (PLEG): container finished" podID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerID="d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738" exitCode=0
Aug 04 18:51:46.415374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.415278    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"
Aug 04 18:51:46.415374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.415290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" event={"ID":"bdca2b4a-d2a2-42ac-bfad-cad320e641fb","Type":"ContainerDied","Data":"d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738"}
Aug 04 18:51:46.415374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.415347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v" event={"ID":"bdca2b4a-d2a2-42ac-bfad-cad320e641fb","Type":"ContainerDied","Data":"e27446473253bbb70d5ca33d0dafc64229a46da3ab6a0ec8ad1e342117b4032b"}
Aug 04 18:51:46.415374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.415369    2581 scope.go:117] "RemoveContainer" containerID="d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738"
Aug 04 18:51:46.424289 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.424268    2581 scope.go:117] "RemoveContainer" containerID="aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4"
Aug 04 18:51:46.431899 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.431882    2581 scope.go:117] "RemoveContainer" containerID="d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738"
Aug 04 18:51:46.432146 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:51:46.432126    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738\": container with ID starting with d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738 not found: ID does not exist" containerID="d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738"
Aug 04 18:51:46.432193 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.432156    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738"} err="failed to get container status \"d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738\": rpc error: code = NotFound desc = could not find container \"d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738\": container with ID starting with d0e8bc030d988cafe7986a1f603d840a393313d973213d91f6459dd6a5259738 not found: ID does not exist"
Aug 04 18:51:46.432193 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.432173    2581 scope.go:117] "RemoveContainer" containerID="aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4"
Aug 04 18:51:46.432391 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:51:46.432371    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4\": container with ID starting with aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4 not found: ID does not exist" containerID="aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4"
Aug 04 18:51:46.432450 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.432401    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4"} err="failed to get container status \"aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4\": rpc error: code = NotFound desc = could not find container \"aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4\": container with ID starting with aa3a0c9fdbfe06bc59a8ca1e5c0333cdb8a52b5396d37550552dad94680260f4 not found: ID does not exist"
Aug 04 18:51:46.439555 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.439529    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"]
Aug 04 18:51:46.445777 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.445759    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-785f6746f-mx77v"]
Aug 04 18:51:46.546382 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:46.546298    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" path="/var/lib/kubelet/pods/bdca2b4a-d2a2-42ac-bfad-cad320e641fb/volumes"
Aug 04 18:51:55.852836 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.852804    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"]
Aug 04 18:51:55.853192 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.853080    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main"
Aug 04 18:51:55.853192 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.853091    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main"
Aug 04 18:51:55.853192 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.853104    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="storage-initializer"
Aug 04 18:51:55.853192 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.853110    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="storage-initializer"
Aug 04 18:51:55.853192 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.853159    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdca2b4a-d2a2-42ac-bfad-cad320e641fb" containerName="main"
Aug 04 18:51:55.857617 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.857590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:55.860589 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.860566    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-7xmv7\""
Aug 04 18:51:55.860939 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.860914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Aug 04 18:51:55.861077 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.860917    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Aug 04 18:51:55.861444 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.861430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:51:55.861514 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.861452    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Aug 04 18:51:55.861573 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.861537    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Aug 04 18:51:55.872689 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:55.872667    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"]
Aug 04 18:51:56.012845 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.012812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acbfea15-316d-421c-bec4-10499ca5195c-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.012996 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.012868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.012996 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.012895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5df66\" (UniqueName: \"kubernetes.io/projected/acbfea15-316d-421c-bec4-10499ca5195c-kube-api-access-5df66\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.012996 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.012915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-model-cache\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.012996 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.012942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acbfea15-316d-421c-bec4-10499ca5195c-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.013156 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.013026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-home\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.013156 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.013055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.013156 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.013075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-dshm\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.113680 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.113571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acbfea15-316d-421c-bec4-10499ca5195c-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.113680 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.113646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.113680 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.113667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5df66\" (UniqueName: \"kubernetes.io/projected/acbfea15-316d-421c-bec4-10499ca5195c-kube-api-access-5df66\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.113937 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.113684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-model-cache\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.113937 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.113701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acbfea15-316d-421c-bec4-10499ca5195c-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.113937 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.113748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-home\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.113937 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.113763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.113937 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.113779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-dshm\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.114241 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.114211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-model-cache\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.114375 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.114245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.114375 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.114263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-home\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.114375 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.114348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.114585 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.114566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acbfea15-316d-421c-bec4-10499ca5195c-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.116036 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.116009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-dshm\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.116279 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.116263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acbfea15-316d-421c-bec4-10499ca5195c-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.122794 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.122774    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5df66\" (UniqueName: \"kubernetes.io/projected/acbfea15-316d-421c-bec4-10499ca5195c-kube-api-access-5df66\") pod \"lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.166929 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.166889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:51:56.301147 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.301118    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"]
Aug 04 18:51:56.303685 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:51:56.303660    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbfea15_316d_421c_bec4_10499ca5195c.slice/crio-ef98788995e84b45830d5ac0603b019a87b3450141cdb11d94e41ab284accba3 WatchSource:0}: Error finding container ef98788995e84b45830d5ac0603b019a87b3450141cdb11d94e41ab284accba3: Status 404 returned error can't find the container with id ef98788995e84b45830d5ac0603b019a87b3450141cdb11d94e41ab284accba3
Aug 04 18:51:56.454508 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.454464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" event={"ID":"acbfea15-316d-421c-bec4-10499ca5195c","Type":"ContainerStarted","Data":"03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd"}
Aug 04 18:51:56.454701 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:51:56.454534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" event={"ID":"acbfea15-316d-421c-bec4-10499ca5195c","Type":"ContainerStarted","Data":"ef98788995e84b45830d5ac0603b019a87b3450141cdb11d94e41ab284accba3"}
Aug 04 18:52:00.471828 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:00.471794    2581 generic.go:356] "Generic (PLEG): container finished" podID="acbfea15-316d-421c-bec4-10499ca5195c" containerID="03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd" exitCode=0
Aug 04 18:52:00.472204 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:00.471866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" event={"ID":"acbfea15-316d-421c-bec4-10499ca5195c","Type":"ContainerDied","Data":"03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd"}
Aug 04 18:52:01.478751 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:01.478709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" event={"ID":"acbfea15-316d-421c-bec4-10499ca5195c","Type":"ContainerStarted","Data":"42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b"}
Aug 04 18:52:02.167615 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:02.167579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:52:06.167921 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:06.167879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:52:06.169721 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:06.169686    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 04 18:52:16.167830 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:16.167732    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 04 18:52:26.168039 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:26.167993    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 04 18:52:36.168057 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:36.168003    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 04 18:52:46.167780 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:46.167725    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 04 18:52:56.168110 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:52:56.168060    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 04 18:53:06.167705 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:53:06.167656    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 04 18:53:16.178258 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:53:16.178212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:53:16.186399 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:53:16.186374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:53:16.204374 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:53:16.204325    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podStartSLOduration=77.919633472 podStartE2EDuration="1m21.20431047s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.284676998s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:52:01.508023027 +0000 UTC m=+3129.442092926" watchObservedRunningTime="2026-08-04 18:53:16.20431047 +0000 UTC m=+3204.138380369"
Aug 04 18:53:43.691922 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:53:43.691887    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb_acbfea15-316d-421c-bec4-10499ca5195c/storage-initializer/0.log"
Aug 04 18:53:43.701294 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:53:43.701270    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb_acbfea15-316d-421c-bec4-10499ca5195c/main/0.log"
Aug 04 18:53:44.821317 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:53:44.821238    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"]
Aug 04 18:53:59.842556 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:53:59.842490    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main" containerID="cri-o://42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b" gracePeriod=45
Aug 04 18:54:00.179789 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.179738    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 04 18:54:00.498329 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.498303    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:54:00.679797 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.679746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-kserve-provision-location\") pod \"acbfea15-316d-421c-bec4-10499ca5195c\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") "
Aug 04 18:54:00.680018 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.679832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-model-cache\") pod \"acbfea15-316d-421c-bec4-10499ca5195c\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") "
Aug 04 18:54:00.680018 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.679853    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-home\") pod \"acbfea15-316d-421c-bec4-10499ca5195c\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") "
Aug 04 18:54:00.680018 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.679900    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acbfea15-316d-421c-bec4-10499ca5195c-tls-certs\") pod \"acbfea15-316d-421c-bec4-10499ca5195c\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") "
Aug 04 18:54:00.680018 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.679916    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5df66\" (UniqueName: \"kubernetes.io/projected/acbfea15-316d-421c-bec4-10499ca5195c-kube-api-access-5df66\") pod \"acbfea15-316d-421c-bec4-10499ca5195c\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") "
Aug 04 18:54:00.680018 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.679937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acbfea15-316d-421c-bec4-10499ca5195c-cabundle-cert\") pod \"acbfea15-316d-421c-bec4-10499ca5195c\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") "
Aug 04 18:54:00.680018 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.679958    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-tmp-dir\") pod \"acbfea15-316d-421c-bec4-10499ca5195c\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") "
Aug 04 18:54:00.680339 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.680022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-dshm\") pod \"acbfea15-316d-421c-bec4-10499ca5195c\" (UID: \"acbfea15-316d-421c-bec4-10499ca5195c\") "
Aug 04 18:54:00.680339 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.680123    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-model-cache" (OuterVolumeSpecName: "model-cache") pod "acbfea15-316d-421c-bec4-10499ca5195c" (UID: "acbfea15-316d-421c-bec4-10499ca5195c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:54:00.680339 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.680237    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.680496 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.680345    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-home" (OuterVolumeSpecName: "home") pod "acbfea15-316d-421c-bec4-10499ca5195c" (UID: "acbfea15-316d-421c-bec4-10499ca5195c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:54:00.680767 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.680728    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbfea15-316d-421c-bec4-10499ca5195c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "acbfea15-316d-421c-bec4-10499ca5195c" (UID: "acbfea15-316d-421c-bec4-10499ca5195c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:54:00.683040 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.683004    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-dshm" (OuterVolumeSpecName: "dshm") pod "acbfea15-316d-421c-bec4-10499ca5195c" (UID: "acbfea15-316d-421c-bec4-10499ca5195c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:54:00.683040 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.683015    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acbfea15-316d-421c-bec4-10499ca5195c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acbfea15-316d-421c-bec4-10499ca5195c" (UID: "acbfea15-316d-421c-bec4-10499ca5195c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:54:00.683040 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.683020    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbfea15-316d-421c-bec4-10499ca5195c-kube-api-access-5df66" (OuterVolumeSpecName: "kube-api-access-5df66") pod "acbfea15-316d-421c-bec4-10499ca5195c" (UID: "acbfea15-316d-421c-bec4-10499ca5195c"). InnerVolumeSpecName "kube-api-access-5df66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:54:00.692971 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.692943    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acbfea15-316d-421c-bec4-10499ca5195c" (UID: "acbfea15-316d-421c-bec4-10499ca5195c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:54:00.705080 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.705056    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "acbfea15-316d-421c-bec4-10499ca5195c" (UID: "acbfea15-316d-421c-bec4-10499ca5195c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:54:00.781691 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.781590    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.781691 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.781640    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.781691 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.781650    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acbfea15-316d-421c-bec4-10499ca5195c-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.781691 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.781662    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5df66\" (UniqueName: \"kubernetes.io/projected/acbfea15-316d-421c-bec4-10499ca5195c-kube-api-access-5df66\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.781691 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.781670    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acbfea15-316d-421c-bec4-10499ca5195c-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.781691 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.781679    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.781691 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.781686    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acbfea15-316d-421c-bec4-10499ca5195c-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.973760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.973724    2581 generic.go:356] "Generic (PLEG): container finished" podID="acbfea15-316d-421c-bec4-10499ca5195c" containerID="42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b" exitCode=0
Aug 04 18:54:00.974199 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.973774    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"
Aug 04 18:54:00.974199 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.973820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" event={"ID":"acbfea15-316d-421c-bec4-10499ca5195c","Type":"ContainerDied","Data":"42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b"}
Aug 04 18:54:00.974199 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.973888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb" event={"ID":"acbfea15-316d-421c-bec4-10499ca5195c","Type":"ContainerDied","Data":"ef98788995e84b45830d5ac0603b019a87b3450141cdb11d94e41ab284accba3"}
Aug 04 18:54:00.974199 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.973925    2581 scope.go:117] "RemoveContainer" containerID="42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b"
Aug 04 18:54:00.982385 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.982365    2581 scope.go:117] "RemoveContainer" containerID="03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd"
Aug 04 18:54:00.989782 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.989765    2581 scope.go:117] "RemoveContainer" containerID="42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b"
Aug 04 18:54:00.990006 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:54:00.989988    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b\": container with ID starting with 42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b not found: ID does not exist" containerID="42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b"
Aug 04 18:54:00.990052 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.990014    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b"} err="failed to get container status \"42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b\": rpc error: code = NotFound desc = could not find container \"42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b\": container with ID starting with 42ee95b00c0854d3abdb22ace99e58d13f6605daed1a8497f287cf7197d9893b not found: ID does not exist"
Aug 04 18:54:00.990052 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.990030    2581 scope.go:117] "RemoveContainer" containerID="03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd"
Aug 04 18:54:00.990242 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:54:00.990224    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd\": container with ID starting with 03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd not found: ID does not exist" containerID="03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd"
Aug 04 18:54:00.990286 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.990245    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd"} err="failed to get container status \"03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd\": rpc error: code = NotFound desc = could not find container \"03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd\": container with ID starting with 03242416963c65e2dfe5d1433f29633baea617761849d6f681310e567c3eacbd not found: ID does not exist"
Aug 04 18:54:00.995706 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.995685    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"]
Aug 04 18:54:00.999899 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:00.999875    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf759dc5d-z26sb"]
Aug 04 18:54:02.545739 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:02.545708    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acbfea15-316d-421c-bec4-10499ca5195c" path="/var/lib/kubelet/pods/acbfea15-316d-421c-bec4-10499ca5195c/volumes"
Aug 04 18:54:11.194155 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.194121    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"]
Aug 04 18:54:11.194519 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.194381    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="storage-initializer"
Aug 04 18:54:11.194519 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.194392    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="storage-initializer"
Aug 04 18:54:11.194519 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.194400    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main"
Aug 04 18:54:11.194519 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.194405    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main"
Aug 04 18:54:11.194519 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.194455    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="acbfea15-316d-421c-bec4-10499ca5195c" containerName="main"
Aug 04 18:54:11.197204 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.197186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.199861 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.199842    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Aug 04 18:54:11.200207 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.200192    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Aug 04 18:54:11.200398 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.200385    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-dv5jt\""
Aug 04 18:54:11.200478 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.200461    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-czcbj\""
Aug 04 18:54:11.201456 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.201431    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:54:11.201584 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.201468    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Aug 04 18:54:11.201584 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.201541    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Aug 04 18:54:11.214551 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.214484    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"]
Aug 04 18:54:11.263827 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.263794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ac63b8-c489-474b-b46f-213cfe3ae3d2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.264009 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.263835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.264009 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.263865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.264009 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.263889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb5pf\" (UniqueName: \"kubernetes.io/projected/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kube-api-access-nb5pf\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.264009 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.263933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.264009 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.263992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.264182 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.264018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.264182 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.264036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.364436 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.364387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ac63b8-c489-474b-b46f-213cfe3ae3d2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.364436 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.364426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.364693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.364461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.364693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.364480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb5pf\" (UniqueName: \"kubernetes.io/projected/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kube-api-access-nb5pf\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.364693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.364502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.364693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.364548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.364693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.364580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.364693 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.364627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.365047 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.365020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.365224 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.365194    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.365303 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.365218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.365303 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.365258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ac63b8-c489-474b-b46f-213cfe3ae3d2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.365455 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.365435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.366813 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.366791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.367310 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.367289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.374670 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.374650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb5pf\" (UniqueName: \"kubernetes.io/projected/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kube-api-access-nb5pf\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.507248 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.507161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:11.645322 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:11.645298    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"]
Aug 04 18:54:11.647882 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:54:11.647851    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ac63b8_c489_474b_b46f_213cfe3ae3d2.slice/crio-abfba49acf1f1552378d8438ff8f8f8fd59eb8a72d0edfc91260307894023cd7 WatchSource:0}: Error finding container abfba49acf1f1552378d8438ff8f8f8fd59eb8a72d0edfc91260307894023cd7: Status 404 returned error can't find the container with id abfba49acf1f1552378d8438ff8f8f8fd59eb8a72d0edfc91260307894023cd7
Aug 04 18:54:12.021454 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:12.021411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" event={"ID":"59ac63b8-c489-474b-b46f-213cfe3ae3d2","Type":"ContainerStarted","Data":"a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f"}
Aug 04 18:54:12.021871 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:12.021497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" event={"ID":"59ac63b8-c489-474b-b46f-213cfe3ae3d2","Type":"ContainerStarted","Data":"abfba49acf1f1552378d8438ff8f8f8fd59eb8a72d0edfc91260307894023cd7"}
Aug 04 18:54:12.021871 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:12.021645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:13.027121 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:13.027081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" event={"ID":"59ac63b8-c489-474b-b46f-213cfe3ae3d2","Type":"ContainerStarted","Data":"b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f"}
Aug 04 18:54:15.037032 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:15.036996    2581 generic.go:356] "Generic (PLEG): container finished" podID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerID="b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f" exitCode=0
Aug 04 18:54:15.037446 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:15.037068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" event={"ID":"59ac63b8-c489-474b-b46f-213cfe3ae3d2","Type":"ContainerDied","Data":"b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f"}
Aug 04 18:54:16.043340 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:16.043302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" event={"ID":"59ac63b8-c489-474b-b46f-213cfe3ae3d2","Type":"ContainerStarted","Data":"6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df"}
Aug 04 18:54:16.074590 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:16.074531    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podStartSLOduration=3.044232158 podStartE2EDuration="5.074514268s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.03028211s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:54:16.073925764 +0000 UTC m=+3264.007995664" watchObservedRunningTime="2026-08-04 18:54:16.074514268 +0000 UTC m=+3264.008584164"
Aug 04 18:54:21.122512 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.122482    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"]
Aug 04 18:54:21.127191 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.127174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.130244 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.130015    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Aug 04 18:54:21.130244 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.130116    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:54:21.130244 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.130130    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-qgkj2\""
Aug 04 18:54:21.130244 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.130125    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Aug 04 18:54:21.130244 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.130164    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Aug 04 18:54:21.130244 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.130193    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Aug 04 18:54:21.138585 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.138561    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"]
Aug 04 18:54:21.247083 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.247048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skzlj\" (UniqueName: \"kubernetes.io/projected/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kube-api-access-skzlj\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.247083 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.247084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d13bafa-d828-421d-ae2d-1b94aa5b2add-cabundle-cert\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.247352 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.247125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kserve-provision-location\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.247352 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.247202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-model-cache\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.247352 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.247254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-home\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.247352 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.247282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-dshm\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.247352 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.247330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tmp-dir\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.247544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.247407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tls-certs\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348125 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tmp-dir\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348125 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tls-certs\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348362 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skzlj\" (UniqueName: \"kubernetes.io/projected/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kube-api-access-skzlj\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348362 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d13bafa-d828-421d-ae2d-1b94aa5b2add-cabundle-cert\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348362 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kserve-provision-location\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348362 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-model-cache\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348571 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-home\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348571 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-dshm\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348696 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tmp-dir\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348758 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-home\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348758 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-model-cache\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.348968 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kserve-provision-location\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.349056 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.348996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d13bafa-d828-421d-ae2d-1b94aa5b2add-cabundle-cert\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.350692 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.350665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-dshm\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.350907 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.350885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tls-certs\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.359598 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.359577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skzlj\" (UniqueName: \"kubernetes.io/projected/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kube-api-access-skzlj\") pod \"tls-verification-test-kserve-6bd4d69484-8r6kd\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.438963 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.438925    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:21.508233 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.508202    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:21.508407 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.508337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:21.509885 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.509819    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:54:21.523465 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.523234    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:54:21.580298 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:21.580267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"]
Aug 04 18:54:21.583459 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:54:21.583434    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d13bafa_d828_421d_ae2d_1b94aa5b2add.slice/crio-5db76d04de4e49ca23dd264c98f42dd4cdc99ddb5577f4b8ca567edbcbc8daf3 WatchSource:0}: Error finding container 5db76d04de4e49ca23dd264c98f42dd4cdc99ddb5577f4b8ca567edbcbc8daf3: Status 404 returned error can't find the container with id 5db76d04de4e49ca23dd264c98f42dd4cdc99ddb5577f4b8ca567edbcbc8daf3
Aug 04 18:54:22.070235 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:22.070186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" event={"ID":"7d13bafa-d828-421d-ae2d-1b94aa5b2add","Type":"ContainerStarted","Data":"bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6"}
Aug 04 18:54:22.070445 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:22.070294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" event={"ID":"7d13bafa-d828-421d-ae2d-1b94aa5b2add","Type":"ContainerStarted","Data":"5db76d04de4e49ca23dd264c98f42dd4cdc99ddb5577f4b8ca567edbcbc8daf3"}
Aug 04 18:54:24.081449 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:24.081412    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerID="bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6" exitCode=0
Aug 04 18:54:24.081992 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:24.081482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" event={"ID":"7d13bafa-d828-421d-ae2d-1b94aa5b2add","Type":"ContainerDied","Data":"bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6"}
Aug 04 18:54:25.088116 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:25.088077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" event={"ID":"7d13bafa-d828-421d-ae2d-1b94aa5b2add","Type":"ContainerStarted","Data":"6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7"}
Aug 04 18:54:25.126051 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:25.125989    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podStartSLOduration=1.896747086 podStartE2EDuration="4.12597054s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.229223454s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:54:25.123410326 +0000 UTC m=+3273.057480224" watchObservedRunningTime="2026-08-04 18:54:25.12597054 +0000 UTC m=+3273.060040438"
Aug 04 18:54:25.439769 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:25.439718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:31.439904 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:31.439859    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:54:31.441400 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:31.441366    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:54:31.508291 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:31.508244    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:54:41.439613 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:41.439551    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:54:41.507707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:41.507662    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:54:51.439781 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:51.439722    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:54:51.508162 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:54:51.508112    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:55:01.440159 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:01.440105    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:55:01.508545 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:01.508491    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:55:11.439351 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:11.439297    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:55:11.507940 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:11.507846    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:55:21.439592 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:21.439488    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:55:21.508218 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:21.508169    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:55:31.439715 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:31.439669    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:55:31.508004 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:31.507953    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:55:41.439893 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:41.439842    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:55:41.508284 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:41.508234    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:55:51.440025 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:51.439972    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:55:51.508429 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:55:51.508375    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:56:01.439878 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:01.439833    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:56:01.507617 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:01.507558    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:56:11.439744 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:11.439687    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:56:11.507879 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:11.507825    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:56:21.439812 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:21.439761    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:56:21.508279 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:21.508221    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:56:31.439351 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:31.439305    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:56:31.508544 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:31.508495    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Aug 04 18:56:41.440283 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:41.440225    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 04 18:56:41.518228 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:41.518193    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:56:41.530394 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:41.530361    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:56:50.444317 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:50.444281    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"]
Aug 04 18:56:51.449349 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:51.449315    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:56:51.456982 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:51.456957    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:56:56.451164 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:56:56.451129    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"]
Aug 04 18:57:05.472744 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:05.472685    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main" containerID="cri-o://6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df" gracePeriod=45
Aug 04 18:57:06.088877 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.088826    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="llm-d-routing-sidecar" containerID="cri-o://a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f" gracePeriod=45
Aug 04 18:57:06.340066 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.340042    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:57:06.466411 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.466379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ac63b8-c489-474b-b46f-213cfe3ae3d2-cabundle-cert\") pod \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") "
Aug 04 18:57:06.466647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.466426    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kserve-provision-location\") pod \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") "
Aug 04 18:57:06.466647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.466454    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb5pf\" (UniqueName: \"kubernetes.io/projected/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kube-api-access-nb5pf\") pod \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") "
Aug 04 18:57:06.466647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.466477    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-home\") pod \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") "
Aug 04 18:57:06.466647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.466510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tls-certs\") pod \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") "
Aug 04 18:57:06.466647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.466577    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tmp-dir\") pod \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") "
Aug 04 18:57:06.466647 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.466626    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-dshm\") pod \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") "
Aug 04 18:57:06.466977 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.466651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-model-cache\") pod \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\" (UID: \"59ac63b8-c489-474b-b46f-213cfe3ae3d2\") "
Aug 04 18:57:06.466977 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.466916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ac63b8-c489-474b-b46f-213cfe3ae3d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "59ac63b8-c489-474b-b46f-213cfe3ae3d2" (UID: "59ac63b8-c489-474b-b46f-213cfe3ae3d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:57:06.467154 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.467106    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "59ac63b8-c489-474b-b46f-213cfe3ae3d2" (UID: "59ac63b8-c489-474b-b46f-213cfe3ae3d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.467320 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.467288    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-home" (OuterVolumeSpecName: "home") pod "59ac63b8-c489-474b-b46f-213cfe3ae3d2" (UID: "59ac63b8-c489-474b-b46f-213cfe3ae3d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.468973 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.468941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59ac63b8-c489-474b-b46f-213cfe3ae3d2" (UID: "59ac63b8-c489-474b-b46f-213cfe3ae3d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:57:06.469066 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.468984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-dshm" (OuterVolumeSpecName: "dshm") pod "59ac63b8-c489-474b-b46f-213cfe3ae3d2" (UID: "59ac63b8-c489-474b-b46f-213cfe3ae3d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.469109 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.469067    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kube-api-access-nb5pf" (OuterVolumeSpecName: "kube-api-access-nb5pf") pod "59ac63b8-c489-474b-b46f-213cfe3ae3d2" (UID: "59ac63b8-c489-474b-b46f-213cfe3ae3d2"). InnerVolumeSpecName "kube-api-access-nb5pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:57:06.479954 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.479916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59ac63b8-c489-474b-b46f-213cfe3ae3d2" (UID: "59ac63b8-c489-474b-b46f-213cfe3ae3d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.489203 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.489179    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59ac63b8-c489-474b-b46f-213cfe3ae3d2" (UID: "59ac63b8-c489-474b-b46f-213cfe3ae3d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.567269 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.567245    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ac63b8-c489-474b-b46f-213cfe3ae3d2-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.567269 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.567269    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.567416 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.567278    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb5pf\" (UniqueName: \"kubernetes.io/projected/59ac63b8-c489-474b-b46f-213cfe3ae3d2-kube-api-access-nb5pf\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.567416 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.567287    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.567416 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.567297    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.567416 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.567305    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.567416 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.567312    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.567416 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.567319    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ac63b8-c489-474b-b46f-213cfe3ae3d2-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.758040 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.758004    2581 generic.go:356] "Generic (PLEG): container finished" podID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerID="6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df" exitCode=0
Aug 04 18:57:06.758040 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.758031    2581 generic.go:356] "Generic (PLEG): container finished" podID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerID="a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f" exitCode=0
Aug 04 18:57:06.758237 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.758066    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"
Aug 04 18:57:06.758237 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.758082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" event={"ID":"59ac63b8-c489-474b-b46f-213cfe3ae3d2","Type":"ContainerDied","Data":"6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df"}
Aug 04 18:57:06.758237 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.758146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" event={"ID":"59ac63b8-c489-474b-b46f-213cfe3ae3d2","Type":"ContainerDied","Data":"a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f"}
Aug 04 18:57:06.758237 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.758186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc" event={"ID":"59ac63b8-c489-474b-b46f-213cfe3ae3d2","Type":"ContainerDied","Data":"abfba49acf1f1552378d8438ff8f8f8fd59eb8a72d0edfc91260307894023cd7"}
Aug 04 18:57:06.758237 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.758207    2581 scope.go:117] "RemoveContainer" containerID="6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df"
Aug 04 18:57:06.766334 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.766318    2581 scope.go:117] "RemoveContainer" containerID="b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f"
Aug 04 18:57:06.773578 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.773556    2581 scope.go:117] "RemoveContainer" containerID="a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f"
Aug 04 18:57:06.780056 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.780033    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"]
Aug 04 18:57:06.780915 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.780894    2581 scope.go:117] "RemoveContainer" containerID="6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df"
Aug 04 18:57:06.781161 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:57:06.781143    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df\": container with ID starting with 6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df not found: ID does not exist" containerID="6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df"
Aug 04 18:57:06.781217 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.781172    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df"} err="failed to get container status \"6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df\": rpc error: code = NotFound desc = could not find container \"6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df\": container with ID starting with 6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df not found: ID does not exist"
Aug 04 18:57:06.781217 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.781195    2581 scope.go:117] "RemoveContainer" containerID="b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f"
Aug 04 18:57:06.781430 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:57:06.781411    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f\": container with ID starting with b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f not found: ID does not exist" containerID="b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f"
Aug 04 18:57:06.781479 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.781433    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f"} err="failed to get container status \"b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f\": rpc error: code = NotFound desc = could not find container \"b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f\": container with ID starting with b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f not found: ID does not exist"
Aug 04 18:57:06.781479 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.781446    2581 scope.go:117] "RemoveContainer" containerID="a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f"
Aug 04 18:57:06.781679 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:57:06.781658    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f\": container with ID starting with a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f not found: ID does not exist" containerID="a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f"
Aug 04 18:57:06.781727 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.781685    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f"} err="failed to get container status \"a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f\": rpc error: code = NotFound desc = could not find container \"a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f\": container with ID starting with a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f not found: ID does not exist"
Aug 04 18:57:06.781727 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.781701    2581 scope.go:117] "RemoveContainer" containerID="6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df"
Aug 04 18:57:06.781942 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.781924    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df"} err="failed to get container status \"6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df\": rpc error: code = NotFound desc = could not find container \"6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df\": container with ID starting with 6713cec54b036ca2ed78c7d0595797040aef4c41a12a0b43b85c98d26cd709df not found: ID does not exist"
Aug 04 18:57:06.781994 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.781943    2581 scope.go:117] "RemoveContainer" containerID="b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f"
Aug 04 18:57:06.782154 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.782133    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f"} err="failed to get container status \"b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f\": rpc error: code = NotFound desc = could not find container \"b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f\": container with ID starting with b261ace875e1f10538010289bb68338c2caf765e4e6194398a1329f9852a6a7f not found: ID does not exist"
Aug 04 18:57:06.782231 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.782156    2581 scope.go:117] "RemoveContainer" containerID="a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f"
Aug 04 18:57:06.782471 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.782428    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f"} err="failed to get container status \"a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f\": rpc error: code = NotFound desc = could not find container \"a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f\": container with ID starting with a2d2c59146aa8506ddb71dcaafca8a3be1296e51bb08a4a2e6b0e10cf4d1c22f not found: ID does not exist"
Aug 04 18:57:06.784617 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:06.784586    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-86959bdf896t2wc"]
Aug 04 18:57:08.546057 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:08.546020    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" path="/var/lib/kubelet/pods/59ac63b8-c489-474b-b46f-213cfe3ae3d2/volumes"
Aug 04 18:57:11.473895 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:11.473848    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main" containerID="cri-o://6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7" gracePeriod=45
Aug 04 18:57:12.216880 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.216856    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:57:12.314364 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kserve-provision-location\") pod \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") "
Aug 04 18:57:12.314364 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314320    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skzlj\" (UniqueName: \"kubernetes.io/projected/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kube-api-access-skzlj\") pod \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") "
Aug 04 18:57:12.314364 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-model-cache\") pod \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") "
Aug 04 18:57:12.314662 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314377    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d13bafa-d828-421d-ae2d-1b94aa5b2add-cabundle-cert\") pod \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") "
Aug 04 18:57:12.314662 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-home\") pod \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") "
Aug 04 18:57:12.314662 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tmp-dir\") pod \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") "
Aug 04 18:57:12.314662 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-dshm\") pod \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") "
Aug 04 18:57:12.314662 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314479    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tls-certs\") pod \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\" (UID: \"7d13bafa-d828-421d-ae2d-1b94aa5b2add\") "
Aug 04 18:57:12.314662 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314646    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d13bafa-d828-421d-ae2d-1b94aa5b2add" (UID: "7d13bafa-d828-421d-ae2d-1b94aa5b2add"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:12.314985 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314754    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:12.314985 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.314846    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d13bafa-d828-421d-ae2d-1b94aa5b2add-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d13bafa-d828-421d-ae2d-1b94aa5b2add" (UID: "7d13bafa-d828-421d-ae2d-1b94aa5b2add"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:57:12.315311 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.315270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-home" (OuterVolumeSpecName: "home") pod "7d13bafa-d828-421d-ae2d-1b94aa5b2add" (UID: "7d13bafa-d828-421d-ae2d-1b94aa5b2add"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:12.316883 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.316851    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-dshm" (OuterVolumeSpecName: "dshm") pod "7d13bafa-d828-421d-ae2d-1b94aa5b2add" (UID: "7d13bafa-d828-421d-ae2d-1b94aa5b2add"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:12.316979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.316904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d13bafa-d828-421d-ae2d-1b94aa5b2add" (UID: "7d13bafa-d828-421d-ae2d-1b94aa5b2add"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:57:12.316979 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.316962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kube-api-access-skzlj" (OuterVolumeSpecName: "kube-api-access-skzlj") pod "7d13bafa-d828-421d-ae2d-1b94aa5b2add" (UID: "7d13bafa-d828-421d-ae2d-1b94aa5b2add"). InnerVolumeSpecName "kube-api-access-skzlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:57:12.334575 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.334550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d13bafa-d828-421d-ae2d-1b94aa5b2add" (UID: "7d13bafa-d828-421d-ae2d-1b94aa5b2add"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:12.336708 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.336688    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d13bafa-d828-421d-ae2d-1b94aa5b2add" (UID: "7d13bafa-d828-421d-ae2d-1b94aa5b2add"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:12.415358 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.415328    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:12.415358 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.415355    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:12.415358 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.415365    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:12.415589 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.415376    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d13bafa-d828-421d-ae2d-1b94aa5b2add-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:12.415589 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.415390    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:12.415589 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.415401    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skzlj\" (UniqueName: \"kubernetes.io/projected/7d13bafa-d828-421d-ae2d-1b94aa5b2add-kube-api-access-skzlj\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:12.415589 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.415409    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d13bafa-d828-421d-ae2d-1b94aa5b2add-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 18:57:12.783295 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.783257    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerID="6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7" exitCode=0
Aug 04 18:57:12.783776 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.783316    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"
Aug 04 18:57:12.783776 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.783327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" event={"ID":"7d13bafa-d828-421d-ae2d-1b94aa5b2add","Type":"ContainerDied","Data":"6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7"}
Aug 04 18:57:12.783776 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.783382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd" event={"ID":"7d13bafa-d828-421d-ae2d-1b94aa5b2add","Type":"ContainerDied","Data":"5db76d04de4e49ca23dd264c98f42dd4cdc99ddb5577f4b8ca567edbcbc8daf3"}
Aug 04 18:57:12.783776 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.783408    2581 scope.go:117] "RemoveContainer" containerID="6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7"
Aug 04 18:57:12.791457 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.791431    2581 scope.go:117] "RemoveContainer" containerID="bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6"
Aug 04 18:57:12.798434 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.798415    2581 scope.go:117] "RemoveContainer" containerID="6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7"
Aug 04 18:57:12.798712 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:57:12.798690    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7\": container with ID starting with 6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7 not found: ID does not exist" containerID="6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7"
Aug 04 18:57:12.798780 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.798718    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7"} err="failed to get container status \"6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7\": rpc error: code = NotFound desc = could not find container \"6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7\": container with ID starting with 6c207cb266c76adce4539b835312758c3580b31a80b576406535ebb246dde5f7 not found: ID does not exist"
Aug 04 18:57:12.798780 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.798735    2581 scope.go:117] "RemoveContainer" containerID="bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6"
Aug 04 18:57:12.798970 ip-10-0-139-201 kubenswrapper[2581]: E0804 18:57:12.798953    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6\": container with ID starting with bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6 not found: ID does not exist" containerID="bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6"
Aug 04 18:57:12.799014 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.798976    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6"} err="failed to get container status \"bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6\": rpc error: code = NotFound desc = could not find container \"bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6\": container with ID starting with bc4e63e359ecf127ce1bf1c4ec36267d6c7dfcdcebc6a39efbe4f164581a29b6 not found: ID does not exist"
Aug 04 18:57:12.821233 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.821206    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"]
Aug 04 18:57:12.828139 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:12.828117    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6bd4d69484-8r6kd"]
Aug 04 18:57:14.544958 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:14.544922    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" path="/var/lib/kubelet/pods/7d13bafa-d828-421d-ae2d-1b94aa5b2add/volumes"
Aug 04 18:57:19.504296 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504267    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"]
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504511    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504522    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504535    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="llm-d-routing-sidecar"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504543    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="llm-d-routing-sidecar"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504557    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="storage-initializer"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504565    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="storage-initializer"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504576    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="storage-initializer"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504582    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="storage-initializer"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504592    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504597    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504658    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="llm-d-routing-sidecar"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504666    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ac63b8-c489-474b-b46f-213cfe3ae3d2" containerName="main"
Aug 04 18:57:19.504707 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.504673    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d13bafa-d828-421d-ae2d-1b94aa5b2add" containerName="main"
Aug 04 18:57:19.509518 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.509498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.512813 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.512778    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Aug 04 18:57:19.513164 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.513150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Aug 04 18:57:19.513255 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.513238    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-w7cdn\""
Aug 04 18:57:19.513504 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.513478    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Aug 04 18:57:19.514028 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.514012    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:57:19.514190 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.514174    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-vglqf\""
Aug 04 18:57:19.517451 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.517428    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Aug 04 18:57:19.527222 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.527198    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"]
Aug 04 18:57:19.677583 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.677550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-dshm\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.677583 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.677588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.677841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.677652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn46f\" (UniqueName: \"kubernetes.io/projected/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kube-api-access-dn46f\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.677841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.677688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-home\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.677841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.677713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.677841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.677735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.677841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.677769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.677841 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.677789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.778997 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.778897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn46f\" (UniqueName: \"kubernetes.io/projected/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kube-api-access-dn46f\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.778997 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.778940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-home\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.778997 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.778960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-dshm\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779280 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-home\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779414    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779559 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.779760 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.779719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.781543 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.781519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-dshm\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.781715 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.781700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.790090 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.790065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn46f\" (UniqueName: \"kubernetes.io/projected/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kube-api-access-dn46f\") pod \"custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:19.818779 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:19.818752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:20.017292 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:20.017265    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"]
Aug 04 18:57:20.018983 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:57:20.018955    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873de0cb_0fb4_4103_8bb5_10b47a82fb7d.slice/crio-80ebfa34e2425ff90f67af138ef1695ef3a00823b56abacaaad51d428ad0fec1 WatchSource:0}: Error finding container 80ebfa34e2425ff90f67af138ef1695ef3a00823b56abacaaad51d428ad0fec1: Status 404 returned error can't find the container with id 80ebfa34e2425ff90f67af138ef1695ef3a00823b56abacaaad51d428ad0fec1
Aug 04 18:57:20.815666 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:20.815632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" event={"ID":"873de0cb-0fb4-4103-8bb5-10b47a82fb7d","Type":"ContainerStarted","Data":"a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01"}
Aug 04 18:57:20.816080 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:20.815686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" event={"ID":"873de0cb-0fb4-4103-8bb5-10b47a82fb7d","Type":"ContainerStarted","Data":"80ebfa34e2425ff90f67af138ef1695ef3a00823b56abacaaad51d428ad0fec1"}
Aug 04 18:57:20.816080 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:20.815739    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:21.821730 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:21.821697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" event={"ID":"873de0cb-0fb4-4103-8bb5-10b47a82fb7d","Type":"ContainerStarted","Data":"cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed"}
Aug 04 18:57:23.831477 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:23.831450    2581 generic.go:356] "Generic (PLEG): container finished" podID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerID="cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed" exitCode=0
Aug 04 18:57:23.831823 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:23.831517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" event={"ID":"873de0cb-0fb4-4103-8bb5-10b47a82fb7d","Type":"ContainerDied","Data":"cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed"}
Aug 04 18:57:24.838164 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:24.838122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" event={"ID":"873de0cb-0fb4-4103-8bb5-10b47a82fb7d","Type":"ContainerStarted","Data":"dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993"}
Aug 04 18:57:24.866365 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:24.866304    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podStartSLOduration=3.89191731 podStartE2EDuration="5.866289401s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.974372091s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:57:24.864708249 +0000 UTC m=+3452.798778149" watchObservedRunningTime="2026-08-04 18:57:24.866289401 +0000 UTC m=+3452.800359300"
Aug 04 18:57:25.016311 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.016279    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"]
Aug 04 18:57:25.019327 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.019311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.023773 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.023746    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Aug 04 18:57:25.024131 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.024109    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:57:25.024263 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.024108    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Aug 04 18:57:25.024263 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.024200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-njgj7\""
Aug 04 18:57:25.024455 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.024437    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Aug 04 18:57:25.024540 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.024501    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Aug 04 18:57:25.036227 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.036202    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"]
Aug 04 18:57:25.130066 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.130032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tls-certs\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.130229 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.130091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-home\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.130229 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.130121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-dshm\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.130229 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.130176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-model-cache\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.130229 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.130210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-cabundle-cert\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.130389 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.130236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txhf\" (UniqueName: \"kubernetes.io/projected/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kube-api-access-5txhf\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.130389 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.130294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kserve-provision-location\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.130389 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.130332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tmp-dir\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.231448 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kserve-provision-location\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.231668 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tmp-dir\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.231668 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tls-certs\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.231668 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-home\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.231668 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-dshm\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.231668 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-model-cache\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.231668 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-cabundle-cert\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.231975 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txhf\" (UniqueName: \"kubernetes.io/projected/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kube-api-access-5txhf\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.231975 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tmp-dir\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.232097 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.231972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kserve-provision-location\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.232272 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.232249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-home\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.232358 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.232267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-model-cache\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.232358 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.232325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-cabundle-cert\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.233963 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.233941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-dshm\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.234372 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.234347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tls-certs\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.242645 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.242596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txhf\" (UniqueName: \"kubernetes.io/projected/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kube-api-access-5txhf\") pod \"prestop-hook-test-kserve-765cd5b55b-dd8mn\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.329809 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.329771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:25.470796 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.470761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"]
Aug 04 18:57:25.474138 ip-10-0-139-201 kubenswrapper[2581]: W0804 18:57:25.474110    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc6e79f_0d77_437f_8a2b_e8d8289b5a70.slice/crio-68c901fd36f904070ab0147fe6ae0709c314cc5f0cb358d452567976ef0c5571 WatchSource:0}: Error finding container 68c901fd36f904070ab0147fe6ae0709c314cc5f0cb358d452567976ef0c5571: Status 404 returned error can't find the container with id 68c901fd36f904070ab0147fe6ae0709c314cc5f0cb358d452567976ef0c5571
Aug 04 18:57:25.844582 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.844487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" event={"ID":"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70","Type":"ContainerStarted","Data":"422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953"}
Aug 04 18:57:25.844582 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:25.844557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" event={"ID":"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70","Type":"ContainerStarted","Data":"68c901fd36f904070ab0147fe6ae0709c314cc5f0cb358d452567976ef0c5571"}
Aug 04 18:57:27.855692 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:27.855655    2581 generic.go:356] "Generic (PLEG): container finished" podID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerID="422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953" exitCode=0
Aug 04 18:57:27.856046 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:27.855741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" event={"ID":"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70","Type":"ContainerDied","Data":"422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953"}
Aug 04 18:57:28.862136 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:28.862085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" event={"ID":"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70","Type":"ContainerStarted","Data":"926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7"}
Aug 04 18:57:28.887198 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:28.887140    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podStartSLOduration=2.716355475 podStartE2EDuration="4.887121914s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.170766439s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:57:28.884880908 +0000 UTC m=+3456.818950833" watchObservedRunningTime="2026-08-04 18:57:28.887121914 +0000 UTC m=+3456.821191813"
Aug 04 18:57:29.330939 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:29.330892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:29.819024 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:29.818981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:29.819233 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:29.819070    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:29.820504 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:29.820475    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:57:35.330098 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:35.330065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 18:57:35.331682 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:35.331655    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:57:39.819423 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:39.819365    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:57:39.836950 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:39.836920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:57:45.330779 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:45.330733    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:57:49.819470 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:49.819419    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:57:55.330283 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:55.330236    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:57:59.820258 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:57:59.820206    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:58:05.331014 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:05.330956    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:58:09.820104 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:09.820052    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:58:15.331042 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:15.330950    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:58:19.820097 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:19.820044    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:58:25.330620 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:25.330560    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:58:29.819184 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:29.819125    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:58:35.330421 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:35.330371    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:58:39.820044 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:39.819977    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:58:45.331059 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:45.331008    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:58:49.819866 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:49.819811    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:58:55.331152 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:55.331109    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:58:59.819412 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:58:59.819353    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:59:05.330988 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:05.330937    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:59:09.819396 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:09.819354    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:59:15.331279 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:15.331236    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:59:19.820094 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:19.820024    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:59:25.330402 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:25.330362    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:59:29.819781 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:29.819728    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:59:35.331001 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:35.330950    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:59:39.819249 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:39.819186    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:59:45.330818 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:45.330722    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:59:49.820109 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:49.820053    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Aug 04 18:59:55.330962 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:55.330917    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 18:59:59.828916 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:59.828881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 18:59:59.845092 ip-10-0-139-201 kubenswrapper[2581]: I0804 18:59:59.845065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 19:00:05.347417 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:05.347379    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 19:00:05.526298 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:05.526268    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 19:00:08.045144 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:08.045098    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"]
Aug 04 19:00:14.128415 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:14.128357    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"]
Aug 04 19:00:23.067333 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.067291    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" containerID="cri-o://926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7" gracePeriod=45
Aug 04 19:00:23.515325 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.515280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Aug 04 19:00:23.718718 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.718687    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 19:00:23.756288 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.756257    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5txhf\" (UniqueName: \"kubernetes.io/projected/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kube-api-access-5txhf\") pod \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") "
Aug 04 19:00:23.756288 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.756293    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tls-certs\") pod \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") "
Aug 04 19:00:23.756549 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.756314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-dshm\") pod \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") "
Aug 04 19:00:23.756549 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.756336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-home\") pod \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") "
Aug 04 19:00:23.756549 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.756376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-model-cache\") pod \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") "
Aug 04 19:00:23.756549 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.756435    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-cabundle-cert\") pod \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") "
Aug 04 19:00:23.756549 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.756468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kserve-provision-location\") pod \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") "
Aug 04 19:00:23.756549 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.756491    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tmp-dir\") pod \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\" (UID: \"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70\") "
Aug 04 19:00:23.756947 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.756922    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" (UID: "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:23.757140 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.757010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" (UID: "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 19:00:23.757788 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.757747    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-home" (OuterVolumeSpecName: "home") pod "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" (UID: "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:23.759095 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.759063    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-dshm" (OuterVolumeSpecName: "dshm") pod "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" (UID: "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:23.759095 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.759077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kube-api-access-5txhf" (OuterVolumeSpecName: "kube-api-access-5txhf") pod "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" (UID: "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70"). InnerVolumeSpecName "kube-api-access-5txhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:00:23.759553 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.759521    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" (UID: "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:00:23.777798 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.777722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" (UID: "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:23.786521 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.786494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" (UID: "cfc6e79f-0d77-437f-8a2b-e8d8289b5a70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:23.857958 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.857923    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:23.857958 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.857951    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:23.857958 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.857961    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:23.858181 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.857971    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:23.858181 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.857981    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5txhf\" (UniqueName: \"kubernetes.io/projected/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-kube-api-access-5txhf\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:23.858181 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.857988    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:23.858181 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.857996    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:23.858181 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:23.858003    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:24.583192 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.583155    2581 generic.go:356] "Generic (PLEG): container finished" podID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerID="926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7" exitCode=0
Aug 04 19:00:24.583641 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.583198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" event={"ID":"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70","Type":"ContainerDied","Data":"926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7"}
Aug 04 19:00:24.583641 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.583207    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"
Aug 04 19:00:24.583641 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.583245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn" event={"ID":"cfc6e79f-0d77-437f-8a2b-e8d8289b5a70","Type":"ContainerDied","Data":"68c901fd36f904070ab0147fe6ae0709c314cc5f0cb358d452567976ef0c5571"}
Aug 04 19:00:24.583641 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.583267    2581 scope.go:117] "RemoveContainer" containerID="926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7"
Aug 04 19:00:24.591361 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.591341    2581 scope.go:117] "RemoveContainer" containerID="422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953"
Aug 04 19:00:24.599071 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.599043    2581 scope.go:117] "RemoveContainer" containerID="926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7"
Aug 04 19:00:24.599353 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:00:24.599332    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7\": container with ID starting with 926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7 not found: ID does not exist" containerID="926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7"
Aug 04 19:00:24.599434 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.599369    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7"} err="failed to get container status \"926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7\": rpc error: code = NotFound desc = could not find container \"926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7\": container with ID starting with 926ec89803400c2c5409fa60926e8317d482f9d93b41fd1fe6d54ce1b2b582f7 not found: ID does not exist"
Aug 04 19:00:24.599434 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.599395    2581 scope.go:117] "RemoveContainer" containerID="422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953"
Aug 04 19:00:24.599639 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:00:24.599620    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953\": container with ID starting with 422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953 not found: ID does not exist" containerID="422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953"
Aug 04 19:00:24.599691 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.599648    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953"} err="failed to get container status \"422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953\": rpc error: code = NotFound desc = could not find container \"422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953\": container with ID starting with 422a2e98019de49aa1065956d01ec851c0d7fb31c5b1e9dbdc988aa058765953 not found: ID does not exist"
Aug 04 19:00:24.604834 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.604809    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"]
Aug 04 19:00:24.608646 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:24.608620    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-dd8mn"]
Aug 04 19:00:26.545291 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:26.545245    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" path="/var/lib/kubelet/pods/cfc6e79f-0d77-437f-8a2b-e8d8289b5a70/volumes"
Aug 04 19:00:29.150186 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:29.150138    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" containerID="cri-o://dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993" gracePeriod=45
Aug 04 19:00:29.766591 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:29.766560    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="llm-d-routing-sidecar" containerID="cri-o://a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01" gracePeriod=45
Aug 04 19:00:29.819786 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:29.819751    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Aug 04 19:00:29.819945 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:29.819754    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Aug 04 19:00:29.829730 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:29.829696    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Aug 04 19:00:30.010786 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.010762    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 19:00:30.107448 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.107406    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kserve-provision-location\") pod \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") "
Aug 04 19:00:30.107448 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.107453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tmp-dir\") pod \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") "
Aug 04 19:00:30.107708 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.107489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-dshm\") pod \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") "
Aug 04 19:00:30.107708 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.107507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tls-certs\") pod \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") "
Aug 04 19:00:30.107708 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.107532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-home\") pod \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") "
Aug 04 19:00:30.107708 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.107553    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-model-cache\") pod \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") "
Aug 04 19:00:30.107708 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.107585    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn46f\" (UniqueName: \"kubernetes.io/projected/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kube-api-access-dn46f\") pod \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") "
Aug 04 19:00:30.107708 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.107673    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-cabundle-cert\") pod \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\" (UID: \"873de0cb-0fb4-4103-8bb5-10b47a82fb7d\") "
Aug 04 19:00:30.108017 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.107892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "873de0cb-0fb4-4103-8bb5-10b47a82fb7d" (UID: "873de0cb-0fb4-4103-8bb5-10b47a82fb7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.108183 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.108160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "873de0cb-0fb4-4103-8bb5-10b47a82fb7d" (UID: "873de0cb-0fb4-4103-8bb5-10b47a82fb7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 19:00:30.108345 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.108305    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-home" (OuterVolumeSpecName: "home") pod "873de0cb-0fb4-4103-8bb5-10b47a82fb7d" (UID: "873de0cb-0fb4-4103-8bb5-10b47a82fb7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.109946 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.109911    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kube-api-access-dn46f" (OuterVolumeSpecName: "kube-api-access-dn46f") pod "873de0cb-0fb4-4103-8bb5-10b47a82fb7d" (UID: "873de0cb-0fb4-4103-8bb5-10b47a82fb7d"). InnerVolumeSpecName "kube-api-access-dn46f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:00:30.110057 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.110002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-dshm" (OuterVolumeSpecName: "dshm") pod "873de0cb-0fb4-4103-8bb5-10b47a82fb7d" (UID: "873de0cb-0fb4-4103-8bb5-10b47a82fb7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.110385 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.110364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "873de0cb-0fb4-4103-8bb5-10b47a82fb7d" (UID: "873de0cb-0fb4-4103-8bb5-10b47a82fb7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:00:30.121910 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.121886    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "873de0cb-0fb4-4103-8bb5-10b47a82fb7d" (UID: "873de0cb-0fb4-4103-8bb5-10b47a82fb7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.131111 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.131082    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "873de0cb-0fb4-4103-8bb5-10b47a82fb7d" (UID: "873de0cb-0fb4-4103-8bb5-10b47a82fb7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.208709 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.208629    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.208709 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.208657    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.208709 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.208668    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.208709 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.208675    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.208709 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.208683    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.208709 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.208692    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn46f\" (UniqueName: \"kubernetes.io/projected/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kube-api-access-dn46f\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.208709 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.208700    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.208709 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.208709    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/873de0cb-0fb4-4103-8bb5-10b47a82fb7d-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.609598 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.609563    2581 generic.go:356] "Generic (PLEG): container finished" podID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerID="dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993" exitCode=0
Aug 04 19:00:30.609598 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.609590    2581 generic.go:356] "Generic (PLEG): container finished" podID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerID="a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01" exitCode=0
Aug 04 19:00:30.609830 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.609633    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"
Aug 04 19:00:30.609830 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.609659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" event={"ID":"873de0cb-0fb4-4103-8bb5-10b47a82fb7d","Type":"ContainerDied","Data":"dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993"}
Aug 04 19:00:30.609830 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.609720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" event={"ID":"873de0cb-0fb4-4103-8bb5-10b47a82fb7d","Type":"ContainerDied","Data":"a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01"}
Aug 04 19:00:30.609830 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.609737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n" event={"ID":"873de0cb-0fb4-4103-8bb5-10b47a82fb7d","Type":"ContainerDied","Data":"80ebfa34e2425ff90f67af138ef1695ef3a00823b56abacaaad51d428ad0fec1"}
Aug 04 19:00:30.609830 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.609758    2581 scope.go:117] "RemoveContainer" containerID="dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993"
Aug 04 19:00:30.617891 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.617873    2581 scope.go:117] "RemoveContainer" containerID="cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed"
Aug 04 19:00:30.624827 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.624811    2581 scope.go:117] "RemoveContainer" containerID="a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01"
Aug 04 19:00:30.629892 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.629870    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"]
Aug 04 19:00:30.632921 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.632893    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5c87cf7fbd-s795n"]
Aug 04 19:00:30.633363 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.633349    2581 scope.go:117] "RemoveContainer" containerID="dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993"
Aug 04 19:00:30.633652 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:00:30.633633    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993\": container with ID starting with dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993 not found: ID does not exist" containerID="dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993"
Aug 04 19:00:30.633741 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.633665    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993"} err="failed to get container status \"dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993\": rpc error: code = NotFound desc = could not find container \"dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993\": container with ID starting with dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993 not found: ID does not exist"
Aug 04 19:00:30.633741 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.633688    2581 scope.go:117] "RemoveContainer" containerID="cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed"
Aug 04 19:00:30.633954 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:00:30.633936    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed\": container with ID starting with cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed not found: ID does not exist" containerID="cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed"
Aug 04 19:00:30.633997 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.633959    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed"} err="failed to get container status \"cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed\": rpc error: code = NotFound desc = could not find container \"cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed\": container with ID starting with cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed not found: ID does not exist"
Aug 04 19:00:30.633997 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.633974    2581 scope.go:117] "RemoveContainer" containerID="a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01"
Aug 04 19:00:30.634199 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:00:30.634182    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01\": container with ID starting with a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01 not found: ID does not exist" containerID="a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01"
Aug 04 19:00:30.634239 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.634204    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01"} err="failed to get container status \"a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01\": rpc error: code = NotFound desc = could not find container \"a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01\": container with ID starting with a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01 not found: ID does not exist"
Aug 04 19:00:30.634239 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.634218    2581 scope.go:117] "RemoveContainer" containerID="dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993"
Aug 04 19:00:30.634422 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.634404    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993"} err="failed to get container status \"dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993\": rpc error: code = NotFound desc = could not find container \"dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993\": container with ID starting with dce64494fa2e32437c3e24a00b13e3c3b4d4cf98f6ca48cccccc14ce1552b993 not found: ID does not exist"
Aug 04 19:00:30.634469 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.634422    2581 scope.go:117] "RemoveContainer" containerID="cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed"
Aug 04 19:00:30.634650 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.634624    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed"} err="failed to get container status \"cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed\": rpc error: code = NotFound desc = could not find container \"cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed\": container with ID starting with cf741c00fcadec8c4db70ea2b40802ec96a711a9ddd04a287a604c87ee7b47ed not found: ID does not exist"
Aug 04 19:00:30.634650 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.634648    2581 scope.go:117] "RemoveContainer" containerID="a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01"
Aug 04 19:00:30.634891 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:30.634872    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01"} err="failed to get container status \"a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01\": rpc error: code = NotFound desc = could not find container \"a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01\": container with ID starting with a717e55301a7d48f32dd306154731d43a3e14c1c4ff2d556923b48a3d809ea01 not found: ID does not exist"
Aug 04 19:00:32.545899 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:32.545857    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" path="/var/lib/kubelet/pods/873de0cb-0fb4-4103-8bb5-10b47a82fb7d/volumes"
Aug 04 19:00:34.517388 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517358    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"]
Aug 04 19:00:34.517795 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517775    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="storage-initializer"
Aug 04 19:00:34.517843 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517799    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="storage-initializer"
Aug 04 19:00:34.517843 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517823    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="storage-initializer"
Aug 04 19:00:34.517843 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517832    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="storage-initializer"
Aug 04 19:00:34.517975 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517842    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main"
Aug 04 19:00:34.517975 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517851    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main"
Aug 04 19:00:34.517975 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517862    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main"
Aug 04 19:00:34.517975 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517870    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main"
Aug 04 19:00:34.517975 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517885    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="llm-d-routing-sidecar"
Aug 04 19:00:34.517975 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517893    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="llm-d-routing-sidecar"
Aug 04 19:00:34.517975 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517972    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="main"
Aug 04 19:00:34.518177 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517983    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc6e79f-0d77-437f-8a2b-e8d8289b5a70" containerName="main"
Aug 04 19:00:34.518177 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.517994    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="873de0cb-0fb4-4103-8bb5-10b47a82fb7d" containerName="llm-d-routing-sidecar"
Aug 04 19:00:34.523499 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.523479    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.526369 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.526343    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Aug 04 19:00:34.526524 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.526418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Aug 04 19:00:34.526524 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.526461    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Aug 04 19:00:34.526717 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.526660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-rnbpg\""
Aug 04 19:00:34.538354 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.538332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"]
Aug 04 19:00:34.644535 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.644502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.644721 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.644553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0620f0f8-aca1-4855-bf27-239431093a89-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.644721 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.644572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.644721 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.644659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.644721 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.644684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.644861 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.644735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.644861 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.644762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.644861 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.644782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0620f0f8-aca1-4855-bf27-239431093a89-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.644861 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.644805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdkf2\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-kube-api-access-xdkf2\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745352 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745352 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0620f0f8-aca1-4855-bf27-239431093a89-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745635 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkf2\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-kube-api-access-xdkf2\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745635 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745635 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0620f0f8-aca1-4855-bf27-239431093a89-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745635 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745635 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745635 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745635 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745996 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745996 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.745996 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.745923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.746103 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.746031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.746154 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.746135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0620f0f8-aca1-4855-bf27-239431093a89-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.748352 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.748321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.748463 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.748446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0620f0f8-aca1-4855-bf27-239431093a89-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.753916 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.753892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.754085 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.754064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdkf2\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-kube-api-access-xdkf2\") pod \"router-gateway-2-openshift-default-6cb79f7856-2nnlh\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.834087 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.833984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:34.966246 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.966222    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"]
Aug 04 19:00:34.968514 ip-10-0-139-201 kubenswrapper[2581]: W0804 19:00:34.968491    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0620f0f8_aca1_4855_bf27_239431093a89.slice/crio-688a6b99a0dde5a5d9913fc99f1d7e6019a1293a0d1d49ee71a96bf85cf4773d WatchSource:0}: Error finding container 688a6b99a0dde5a5d9913fc99f1d7e6019a1293a0d1d49ee71a96bf85cf4773d: Status 404 returned error can't find the container with id 688a6b99a0dde5a5d9913fc99f1d7e6019a1293a0d1d49ee71a96bf85cf4773d
Aug 04 19:00:34.970593 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.970566    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 19:00:34.970670 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.970646    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 19:00:34.970712 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:34.970681    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 19:00:35.629403 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:35.629360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" event={"ID":"0620f0f8-aca1-4855-bf27-239431093a89","Type":"ContainerStarted","Data":"07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180"}
Aug 04 19:00:35.629821 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:35.629427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" event={"ID":"0620f0f8-aca1-4855-bf27-239431093a89","Type":"ContainerStarted","Data":"688a6b99a0dde5a5d9913fc99f1d7e6019a1293a0d1d49ee71a96bf85cf4773d"}
Aug 04 19:00:35.652508 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:35.652460    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podStartSLOduration=1.652444783 podStartE2EDuration="1.652444783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:00:35.651384766 +0000 UTC m=+3643.585454665" watchObservedRunningTime="2026-08-04 19:00:35.652444783 +0000 UTC m=+3643.586514759"
Aug 04 19:00:35.834246 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:35.834208    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:37.836078 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:37.836028    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 19:00:38.837427 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:38.837309    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 19:00:39.838064 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:39.838013    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 19:00:40.838826 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:40.838780    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 19:00:41.840070 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:41.840030    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 19:00:42.880843 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:42.880771    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 19:00:43.922788 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:43.922739    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 19:00:44.362921 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:44.362879    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:42030->10.133.0.51:15021: read: connection reset by peer"
Aug 04 19:00:44.680374 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:44.680344    2581 generic.go:356] "Generic (PLEG): container finished" podID="0620f0f8-aca1-4855-bf27-239431093a89" containerID="07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180" exitCode=0
Aug 04 19:00:44.680537 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:44.680421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" event={"ID":"0620f0f8-aca1-4855-bf27-239431093a89","Type":"ContainerDied","Data":"07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180"}
Aug 04 19:00:44.680816 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:44.680803    2581 scope.go:117] "RemoveContainer" containerID="07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180"
Aug 04 19:00:44.681792 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:44.681764    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 19:00:44.681857 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:44.681832    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 19:00:44.681893 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:44.681872    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 19:00:44.834906 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:44.834867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:45.686625 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:45.686573    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" event={"ID":"0620f0f8-aca1-4855-bf27-239431093a89","Type":"ContainerStarted","Data":"1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c"}
Aug 04 19:00:45.835263 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:45.835210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:45.835545 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:45.835296    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Aug 04 19:00:46.341218 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.341183    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"]
Aug 04 19:00:46.344656 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.344632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.348922 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.348892    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Aug 04 19:00:46.349212 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.349193    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 19:00:46.349518 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.349194    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-zkkbr\""
Aug 04 19:00:46.349913 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.349893    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-m65hb\""
Aug 04 19:00:46.350263 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.350227    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Aug 04 19:00:46.366625 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.364542    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"]
Aug 04 19:00:46.367175 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.367153    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"]
Aug 04 19:00:46.370076 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.370054    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.390158 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.390134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"]
Aug 04 19:00:46.448052 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.448000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.448355 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.448329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.448501 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.448481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrbds\" (UniqueName: \"kubernetes.io/projected/0436453c-3113-4ea4-8ce8-0ddedefd115f-kube-api-access-wrbds\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.448662 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.448644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0436453c-3113-4ea4-8ce8-0ddedefd115f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.448803 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.448784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.448930 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.448914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-home\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.449039 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.449025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.449166 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.449150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.449283 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.449266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0436453c-3113-4ea4-8ce8-0ddedefd115f-tls-certs\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.449389 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.449375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-home\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.449508 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.449484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9998b74-f323-4f23-a047-cb55c566e16d-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.449658 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.449641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjt66\" (UniqueName: \"kubernetes.io/projected/e9998b74-f323-4f23-a047-cb55c566e16d-kube-api-access-rjt66\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.449789 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.449774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-model-cache\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.449932 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.449916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-dshm\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.450038 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.450024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9998b74-f323-4f23-a047-cb55c566e16d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.450146 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.450131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.550675 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.550867 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0436453c-3113-4ea4-8ce8-0ddedefd115f-tls-certs\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.550867 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-home\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.550867 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9998b74-f323-4f23-a047-cb55c566e16d-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.550867 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjt66\" (UniqueName: \"kubernetes.io/projected/e9998b74-f323-4f23-a047-cb55c566e16d-kube-api-access-rjt66\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.550867 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-model-cache\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.550867 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-dshm\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.551208 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9998b74-f323-4f23-a047-cb55c566e16d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.551208 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.551208 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.551208 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.550978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.551208 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.551016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrbds\" (UniqueName: \"kubernetes.io/projected/0436453c-3113-4ea4-8ce8-0ddedefd115f-kube-api-access-wrbds\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.551208 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.551044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0436453c-3113-4ea4-8ce8-0ddedefd115f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.551208 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.551075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.551208 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.551111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-home\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.551208 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.551137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.552446 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.552422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.553937 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.553031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.554109 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.553276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9998b74-f323-4f23-a047-cb55c566e16d-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.554251 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.554222    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0436453c-3113-4ea4-8ce8-0ddedefd115f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.554339 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.553286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-model-cache\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.554339 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.553512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-home\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.554339 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.553570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.554339 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.553842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-home\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.554544 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.553277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.554930 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.554909    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.557632 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.557373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0436453c-3113-4ea4-8ce8-0ddedefd115f-tls-certs\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.560194 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.559840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-dshm\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.560194 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.560022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.560194 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.560163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9998b74-f323-4f23-a047-cb55c566e16d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.566047 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.566024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrbds\" (UniqueName: \"kubernetes.io/projected/0436453c-3113-4ea4-8ce8-0ddedefd115f-kube-api-access-wrbds\") pod \"router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.567503 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.567481    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjt66\" (UniqueName: \"kubernetes.io/projected/e9998b74-f323-4f23-a047-cb55c566e16d-kube-api-access-rjt66\") pod \"router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.660399 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.660362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:46.682778 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.682748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:46.859155 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.859106    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Aug 04 19:00:46.860499 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.860466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"]
Aug 04 19:00:46.862105 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:46.862082    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"]
Aug 04 19:00:46.863065 ip-10-0-139-201 kubenswrapper[2581]: W0804 19:00:46.863036    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0436453c_3113_4ea4_8ce8_0ddedefd115f.slice/crio-0b92f9022b3ade18a6ea8020e11d8d3096d15a4b218f25cfb9a927274bd1a83c WatchSource:0}: Error finding container 0b92f9022b3ade18a6ea8020e11d8d3096d15a4b218f25cfb9a927274bd1a83c: Status 404 returned error can't find the container with id 0b92f9022b3ade18a6ea8020e11d8d3096d15a4b218f25cfb9a927274bd1a83c
Aug 04 19:00:46.866957 ip-10-0-139-201 kubenswrapper[2581]: W0804 19:00:46.866937    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9998b74_f323_4f23_a047_cb55c566e16d.slice/crio-479631c3d4853116ca24c4e97aa39a625571123ba6ec653d7e7d1de6c18a0b78 WatchSource:0}: Error finding container 479631c3d4853116ca24c4e97aa39a625571123ba6ec653d7e7d1de6c18a0b78: Status 404 returned error can't find the container with id 479631c3d4853116ca24c4e97aa39a625571123ba6ec653d7e7d1de6c18a0b78
Aug 04 19:00:47.696673 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:47.696624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" event={"ID":"0436453c-3113-4ea4-8ce8-0ddedefd115f","Type":"ContainerStarted","Data":"b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e"}
Aug 04 19:00:47.696882 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:47.696694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" event={"ID":"0436453c-3113-4ea4-8ce8-0ddedefd115f","Type":"ContainerStarted","Data":"0b92f9022b3ade18a6ea8020e11d8d3096d15a4b218f25cfb9a927274bd1a83c"}
Aug 04 19:00:47.696882 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:47.696726    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:47.698211 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:47.698184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" event={"ID":"e9998b74-f323-4f23-a047-cb55c566e16d","Type":"ContainerStarted","Data":"700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632"}
Aug 04 19:00:47.698309 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:47.698238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" event={"ID":"e9998b74-f323-4f23-a047-cb55c566e16d","Type":"ContainerStarted","Data":"479631c3d4853116ca24c4e97aa39a625571123ba6ec653d7e7d1de6c18a0b78"}
Aug 04 19:00:47.835141 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:47.835101    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Aug 04 19:00:48.703864 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:48.703826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" event={"ID":"0436453c-3113-4ea4-8ce8-0ddedefd115f","Type":"ContainerStarted","Data":"2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e"}
Aug 04 19:00:48.834731 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:48.834681    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Aug 04 19:00:49.711345 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:49.711265    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9998b74-f323-4f23-a047-cb55c566e16d" containerID="700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632" exitCode=0
Aug 04 19:00:49.711708 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:49.711538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" event={"ID":"e9998b74-f323-4f23-a047-cb55c566e16d","Type":"ContainerDied","Data":"700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632"}
Aug 04 19:00:49.835062 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:49.835014    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Aug 04 19:00:49.835496 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:49.835472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:50.717575 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:50.717528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" event={"ID":"e9998b74-f323-4f23-a047-cb55c566e16d","Type":"ContainerStarted","Data":"320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee"}
Aug 04 19:00:50.719409 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:50.719378    2581 generic.go:356] "Generic (PLEG): container finished" podID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerID="2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e" exitCode=0
Aug 04 19:00:50.719517 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:50.719442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" event={"ID":"0436453c-3113-4ea4-8ce8-0ddedefd115f","Type":"ContainerDied","Data":"2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e"}
Aug 04 19:00:50.744517 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:50.744453    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podStartSLOduration=2.254766471 podStartE2EDuration="4.74443037s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.489663899s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:00:50.741737623 +0000 UTC m=+3658.675807524" watchObservedRunningTime="2026-08-04 19:00:50.74443037 +0000 UTC m=+3658.678500271"
Aug 04 19:00:50.934428 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:50.934250    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:50.935820 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:50.935788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:00:51.726388 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:51.726348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" event={"ID":"0436453c-3113-4ea4-8ce8-0ddedefd115f","Type":"ContainerStarted","Data":"ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688"}
Aug 04 19:00:51.755988 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:51.755932    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podStartSLOduration=3.270525502 podStartE2EDuration="5.755913556s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.485388054s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:00:51.753140537 +0000 UTC m=+3659.687210436" watchObservedRunningTime="2026-08-04 19:00:51.755913556 +0000 UTC m=+3659.689983454"
Aug 04 19:00:56.661318 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:56.661274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:56.661904 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:56.661353    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:00:56.662949 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:56.662806    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:00:56.683025 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:56.682977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:56.683225 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:56.683102    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:00:56.684414 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:56.684385    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:00:56.687551 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:00:56.687528    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:01:06.661742 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:06.661679    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:01:06.683855 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:06.683815    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:01:16.661563 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:16.661449    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:01:16.683720 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:16.683680    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:01:26.661360 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:26.661306    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:01:26.683513 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:26.683461    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:01:36.661288 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:36.661224    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:01:36.683667 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:36.683629    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:01:46.660905 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:46.660844    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:01:46.683396 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:46.683352    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:01:56.660944 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:56.660896    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:01:56.683616 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:01:56.683556    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:02:06.661519 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:06.661462    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:02:06.683230 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:06.683184    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:02:16.661078 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:16.661018    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:02:16.683419 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:16.683374    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:02:26.661045 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:26.660996    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:02:26.684058 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:26.684018    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:02:36.661132 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:36.661079    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:02:36.683867 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:36.683832    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:02:46.661254 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:46.661143    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:02:46.683791 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:46.683752    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:02:56.661143 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:56.661089    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:02:56.684042 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:02:56.684008    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:03:06.661205 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:06.661160    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:03:06.683156 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:06.683101    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:03:16.660881 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:16.660837    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Aug 04 19:03:16.683786 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:16.683748    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 04 19:03:26.671377 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:26.671342    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:03:26.682926 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:26.682897    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:03:26.692540 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:26.692513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:03:26.700626 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:26.700586    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:03:35.305518 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:35.305478    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"]
Aug 04 19:03:35.316479 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:35.316449    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"]
Aug 04 19:03:50.331176 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:50.331129    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main" containerID="cri-o://ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688" gracePeriod=45
Aug 04 19:03:50.337762 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:50.337726    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main" containerID="cri-o://320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee" gracePeriod=45
Aug 04 19:03:51.143465 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.143430    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="llm-d-routing-sidecar" containerID="cri-o://b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e" gracePeriod=45
Aug 04 19:03:51.280998 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.280968    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:03:51.373419 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.373383    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-home\") pod \"e9998b74-f323-4f23-a047-cb55c566e16d\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") "
Aug 04 19:03:51.373847 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.373441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9998b74-f323-4f23-a047-cb55c566e16d-cabundle-cert\") pod \"e9998b74-f323-4f23-a047-cb55c566e16d\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") "
Aug 04 19:03:51.373847 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.373521    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-model-cache\") pod \"e9998b74-f323-4f23-a047-cb55c566e16d\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") "
Aug 04 19:03:51.373847 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.373562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9998b74-f323-4f23-a047-cb55c566e16d-tls-certs\") pod \"e9998b74-f323-4f23-a047-cb55c566e16d\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") "
Aug 04 19:03:51.373847 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.373612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjt66\" (UniqueName: \"kubernetes.io/projected/e9998b74-f323-4f23-a047-cb55c566e16d-kube-api-access-rjt66\") pod \"e9998b74-f323-4f23-a047-cb55c566e16d\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") "
Aug 04 19:03:51.373847 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.373660    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-kserve-provision-location\") pod \"e9998b74-f323-4f23-a047-cb55c566e16d\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") "
Aug 04 19:03:51.373847 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.373695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-tmp-dir\") pod \"e9998b74-f323-4f23-a047-cb55c566e16d\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") "
Aug 04 19:03:51.373847 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.373717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-dshm\") pod \"e9998b74-f323-4f23-a047-cb55c566e16d\" (UID: \"e9998b74-f323-4f23-a047-cb55c566e16d\") "
Aug 04 19:03:51.374234 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.373976    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9998b74-f323-4f23-a047-cb55c566e16d" (UID: "e9998b74-f323-4f23-a047-cb55c566e16d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.374234 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.374120    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-home" (OuterVolumeSpecName: "home") pod "e9998b74-f323-4f23-a047-cb55c566e16d" (UID: "e9998b74-f323-4f23-a047-cb55c566e16d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.375769 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.375711    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9998b74-f323-4f23-a047-cb55c566e16d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e9998b74-f323-4f23-a047-cb55c566e16d" (UID: "e9998b74-f323-4f23-a047-cb55c566e16d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 19:03:51.376161 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.376138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9998b74-f323-4f23-a047-cb55c566e16d-kube-api-access-rjt66" (OuterVolumeSpecName: "kube-api-access-rjt66") pod "e9998b74-f323-4f23-a047-cb55c566e16d" (UID: "e9998b74-f323-4f23-a047-cb55c566e16d"). InnerVolumeSpecName "kube-api-access-rjt66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:03:51.377152 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.377133    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9998b74-f323-4f23-a047-cb55c566e16d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9998b74-f323-4f23-a047-cb55c566e16d" (UID: "e9998b74-f323-4f23-a047-cb55c566e16d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:03:51.377265 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.377233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-dshm" (OuterVolumeSpecName: "dshm") pod "e9998b74-f323-4f23-a047-cb55c566e16d" (UID: "e9998b74-f323-4f23-a047-cb55c566e16d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.387887 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.387848    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9998b74-f323-4f23-a047-cb55c566e16d" (UID: "e9998b74-f323-4f23-a047-cb55c566e16d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.396828 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.396800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9998b74-f323-4f23-a047-cb55c566e16d" (UID: "e9998b74-f323-4f23-a047-cb55c566e16d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.397004 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.396987    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:03:51.474512 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrbds\" (UniqueName: \"kubernetes.io/projected/0436453c-3113-4ea4-8ce8-0ddedefd115f-kube-api-access-wrbds\") pod \"0436453c-3113-4ea4-8ce8-0ddedefd115f\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") "
Aug 04 19:03:51.474512 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-kserve-provision-location\") pod \"0436453c-3113-4ea4-8ce8-0ddedefd115f\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") "
Aug 04 19:03:51.474778 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-dshm\") pod \"0436453c-3113-4ea4-8ce8-0ddedefd115f\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") "
Aug 04 19:03:51.474778 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0436453c-3113-4ea4-8ce8-0ddedefd115f-cabundle-cert\") pod \"0436453c-3113-4ea4-8ce8-0ddedefd115f\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") "
Aug 04 19:03:51.474778 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-tmp-dir\") pod \"0436453c-3113-4ea4-8ce8-0ddedefd115f\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") "
Aug 04 19:03:51.474778 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-home\") pod \"0436453c-3113-4ea4-8ce8-0ddedefd115f\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") "
Aug 04 19:03:51.474778 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0436453c-3113-4ea4-8ce8-0ddedefd115f-tls-certs\") pod \"0436453c-3113-4ea4-8ce8-0ddedefd115f\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") "
Aug 04 19:03:51.474778 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474714    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-model-cache\") pod \"0436453c-3113-4ea4-8ce8-0ddedefd115f\" (UID: \"0436453c-3113-4ea4-8ce8-0ddedefd115f\") "
Aug 04 19:03:51.475100 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474911    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.475100 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474930    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9998b74-f323-4f23-a047-cb55c566e16d-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.475100 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474948    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.475100 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474961    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9998b74-f323-4f23-a047-cb55c566e16d-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.475100 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474974    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjt66\" (UniqueName: \"kubernetes.io/projected/e9998b74-f323-4f23-a047-cb55c566e16d-kube-api-access-rjt66\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.475100 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.474988    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.475100 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.475000    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.475100 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.475012    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9998b74-f323-4f23-a047-cb55c566e16d-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.475481 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.475217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0436453c-3113-4ea4-8ce8-0ddedefd115f" (UID: "0436453c-3113-4ea4-8ce8-0ddedefd115f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.475538 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.475500    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0436453c-3113-4ea4-8ce8-0ddedefd115f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0436453c-3113-4ea4-8ce8-0ddedefd115f" (UID: "0436453c-3113-4ea4-8ce8-0ddedefd115f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 19:03:51.475594 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.475531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-home" (OuterVolumeSpecName: "home") pod "0436453c-3113-4ea4-8ce8-0ddedefd115f" (UID: "0436453c-3113-4ea4-8ce8-0ddedefd115f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.477264 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.477229    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0436453c-3113-4ea4-8ce8-0ddedefd115f-kube-api-access-wrbds" (OuterVolumeSpecName: "kube-api-access-wrbds") pod "0436453c-3113-4ea4-8ce8-0ddedefd115f" (UID: "0436453c-3113-4ea4-8ce8-0ddedefd115f"). InnerVolumeSpecName "kube-api-access-wrbds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:03:51.477264 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.477234    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-dshm" (OuterVolumeSpecName: "dshm") pod "0436453c-3113-4ea4-8ce8-0ddedefd115f" (UID: "0436453c-3113-4ea4-8ce8-0ddedefd115f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.477487 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.477470    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0436453c-3113-4ea4-8ce8-0ddedefd115f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0436453c-3113-4ea4-8ce8-0ddedefd115f" (UID: "0436453c-3113-4ea4-8ce8-0ddedefd115f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:03:51.481785 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.481756    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9998b74-f323-4f23-a047-cb55c566e16d" containerID="320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee" exitCode=0
Aug 04 19:03:51.481910 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.481813    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"
Aug 04 19:03:51.481910 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.481829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" event={"ID":"e9998b74-f323-4f23-a047-cb55c566e16d","Type":"ContainerDied","Data":"320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee"}
Aug 04 19:03:51.481910 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.481885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87" event={"ID":"e9998b74-f323-4f23-a047-cb55c566e16d","Type":"ContainerDied","Data":"479631c3d4853116ca24c4e97aa39a625571123ba6ec653d7e7d1de6c18a0b78"}
Aug 04 19:03:51.482076 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.481920    2581 scope.go:117] "RemoveContainer" containerID="320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee"
Aug 04 19:03:51.491507 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.491478    2581 generic.go:356] "Generic (PLEG): container finished" podID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerID="ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688" exitCode=0
Aug 04 19:03:51.491662 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.491510    2581 generic.go:356] "Generic (PLEG): container finished" podID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerID="b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e" exitCode=0
Aug 04 19:03:51.491662 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.491534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" event={"ID":"0436453c-3113-4ea4-8ce8-0ddedefd115f","Type":"ContainerDied","Data":"ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688"}
Aug 04 19:03:51.491662 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.491643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" event={"ID":"0436453c-3113-4ea4-8ce8-0ddedefd115f","Type":"ContainerDied","Data":"b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e"}
Aug 04 19:03:51.491833 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.491676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g" event={"ID":"0436453c-3113-4ea4-8ce8-0ddedefd115f","Type":"ContainerDied","Data":"0b92f9022b3ade18a6ea8020e11d8d3096d15a4b218f25cfb9a927274bd1a83c"}
Aug 04 19:03:51.491881 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.491861    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"
Aug 04 19:03:51.494327 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.494297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0436453c-3113-4ea4-8ce8-0ddedefd115f" (UID: "0436453c-3113-4ea4-8ce8-0ddedefd115f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.498687 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.498619    2581 scope.go:117] "RemoveContainer" containerID="700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632"
Aug 04 19:03:51.506332 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.506315    2581 scope.go:117] "RemoveContainer" containerID="320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee"
Aug 04 19:03:51.506577 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.506554    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0436453c-3113-4ea4-8ce8-0ddedefd115f" (UID: "0436453c-3113-4ea4-8ce8-0ddedefd115f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:03:51.506665 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:03:51.506573    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee\": container with ID starting with 320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee not found: ID does not exist" containerID="320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee"
Aug 04 19:03:51.506665 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.506613    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee"} err="failed to get container status \"320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee\": rpc error: code = NotFound desc = could not find container \"320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee\": container with ID starting with 320ad2406b783931e476533be97fde3d06c86e65d644541606321a8acc2e28ee not found: ID does not exist"
Aug 04 19:03:51.506665 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.506634    2581 scope.go:117] "RemoveContainer" containerID="700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632"
Aug 04 19:03:51.506868 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:03:51.506848    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632\": container with ID starting with 700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632 not found: ID does not exist" containerID="700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632"
Aug 04 19:03:51.506907 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.506875    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632"} err="failed to get container status \"700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632\": rpc error: code = NotFound desc = could not find container \"700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632\": container with ID starting with 700fddc1995aff88911dd9c32c14f2d5fc4b9865e7612569c13d0f18ada82632 not found: ID does not exist"
Aug 04 19:03:51.506907 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.506892    2581 scope.go:117] "RemoveContainer" containerID="ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688"
Aug 04 19:03:51.514872 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.514848    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"]
Aug 04 19:03:51.515117 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.515102    2581 scope.go:117] "RemoveContainer" containerID="2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e"
Aug 04 19:03:51.517899 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.517879    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-65558d8d49-msg87"]
Aug 04 19:03:51.522494 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.522479    2581 scope.go:117] "RemoveContainer" containerID="b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e"
Aug 04 19:03:51.529040 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.529023    2581 scope.go:117] "RemoveContainer" containerID="ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688"
Aug 04 19:03:51.529271 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:03:51.529254    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688\": container with ID starting with ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688 not found: ID does not exist" containerID="ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688"
Aug 04 19:03:51.529318 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.529278    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688"} err="failed to get container status \"ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688\": rpc error: code = NotFound desc = could not find container \"ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688\": container with ID starting with ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688 not found: ID does not exist"
Aug 04 19:03:51.529318 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.529294    2581 scope.go:117] "RemoveContainer" containerID="2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e"
Aug 04 19:03:51.529517 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:03:51.529501    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e\": container with ID starting with 2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e not found: ID does not exist" containerID="2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e"
Aug 04 19:03:51.529567 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.529521    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e"} err="failed to get container status \"2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e\": rpc error: code = NotFound desc = could not find container \"2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e\": container with ID starting with 2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e not found: ID does not exist"
Aug 04 19:03:51.529567 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.529533    2581 scope.go:117] "RemoveContainer" containerID="b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e"
Aug 04 19:03:51.529743 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:03:51.529727    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e\": container with ID starting with b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e not found: ID does not exist" containerID="b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e"
Aug 04 19:03:51.529787 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.529746    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e"} err="failed to get container status \"b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e\": rpc error: code = NotFound desc = could not find container \"b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e\": container with ID starting with b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e not found: ID does not exist"
Aug 04 19:03:51.529787 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.529759    2581 scope.go:117] "RemoveContainer" containerID="ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688"
Aug 04 19:03:51.529937 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.529921    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688"} err="failed to get container status \"ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688\": rpc error: code = NotFound desc = could not find container \"ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688\": container with ID starting with ab4aeb413f905d3875d366461f1ed3fc9b8741b873009712f425ba1852969688 not found: ID does not exist"
Aug 04 19:03:51.529979 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.529937    2581 scope.go:117] "RemoveContainer" containerID="2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e"
Aug 04 19:03:51.530125 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.530109    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e"} err="failed to get container status \"2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e\": rpc error: code = NotFound desc = could not find container \"2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e\": container with ID starting with 2c45d964ed955f5e5a04a8a9d93aa5e2abaf43c1fd86ae0af76b1ebaea9f688e not found: ID does not exist"
Aug 04 19:03:51.530169 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.530125    2581 scope.go:117] "RemoveContainer" containerID="b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e"
Aug 04 19:03:51.530362 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.530341    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e"} err="failed to get container status \"b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e\": rpc error: code = NotFound desc = could not find container \"b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e\": container with ID starting with b7d8204f8ba850bcb51f523b4ac67fc57d7b81c3d68b5b33943d4156dfd4e05e not found: ID does not exist"
Aug 04 19:03:51.576099 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.576023    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0436453c-3113-4ea4-8ce8-0ddedefd115f-cabundle-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.576099 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.576078    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.576099 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.576091    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.576279 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.576104    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0436453c-3113-4ea4-8ce8-0ddedefd115f-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.576279 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.576116    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.576279 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.576127    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrbds\" (UniqueName: \"kubernetes.io/projected/0436453c-3113-4ea4-8ce8-0ddedefd115f-kube-api-access-wrbds\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.576279 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.576140    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-kserve-provision-location\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.576279 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.576152    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0436453c-3113-4ea4-8ce8-0ddedefd115f-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:03:51.816135 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.816107    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"]
Aug 04 19:03:51.822907 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:51.822881    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-58c4fb8fd8-fkc2g"]
Aug 04 19:03:52.546927 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:52.546890    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" path="/var/lib/kubelet/pods/0436453c-3113-4ea4-8ce8-0ddedefd115f/volumes"
Aug 04 19:03:52.547395 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:03:52.547382    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" path="/var/lib/kubelet/pods/e9998b74-f323-4f23-a047-cb55c566e16d/volumes"
Aug 04 19:04:00.314304 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:00.314261    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"]
Aug 04 19:04:00.314776 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:00.314488    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy" containerID="cri-o://1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c" gracePeriod=30
Aug 04 19:04:05.546414 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.546389    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:04:05.546803 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.546559    2581 generic.go:356] "Generic (PLEG): container finished" podID="0620f0f8-aca1-4855-bf27-239431093a89" containerID="1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c" exitCode=0
Aug 04 19:04:05.546803 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.546631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" event={"ID":"0620f0f8-aca1-4855-bf27-239431093a89","Type":"ContainerDied","Data":"1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c"}
Aug 04 19:04:05.546803 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.546680    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh" event={"ID":"0620f0f8-aca1-4855-bf27-239431093a89","Type":"ContainerDied","Data":"688a6b99a0dde5a5d9913fc99f1d7e6019a1293a0d1d49ee71a96bf85cf4773d"}
Aug 04 19:04:05.546803 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.546703    2581 scope.go:117] "RemoveContainer" containerID="1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c"
Aug 04 19:04:05.554137 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.554113    2581 scope.go:117] "RemoveContainer" containerID="07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180"
Aug 04 19:04:05.561016 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.560989    2581 scope.go:117] "RemoveContainer" containerID="1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c"
Aug 04 19:04:05.561259 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:04:05.561240    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c\": container with ID starting with 1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c not found: ID does not exist" containerID="1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c"
Aug 04 19:04:05.561300 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.561268    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c"} err="failed to get container status \"1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c\": rpc error: code = NotFound desc = could not find container \"1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c\": container with ID starting with 1ddbf34362b4b9ef139fa971999d1faa8d33bb569d2d5009785d61ee851f9a0c not found: ID does not exist"
Aug 04 19:04:05.561300 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.561285    2581 scope.go:117] "RemoveContainer" containerID="07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180"
Aug 04 19:04:05.561521 ip-10-0-139-201 kubenswrapper[2581]: E0804 19:04:05.561500    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180\": container with ID starting with 07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180 not found: ID does not exist" containerID="07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180"
Aug 04 19:04:05.561563 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.561530    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180"} err="failed to get container status \"07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180\": rpc error: code = NotFound desc = could not find container \"07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180\": container with ID starting with 07e9fd39048852c3b485a931bb6af5284b9950780840925f440feed41b31c180 not found: ID does not exist"
Aug 04 19:04:05.696940 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.696905    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-istio-token\") pod \"0620f0f8-aca1-4855-bf27-239431093a89\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") "
Aug 04 19:04:05.697150 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.696958    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0620f0f8-aca1-4855-bf27-239431093a89-istiod-ca-cert\") pod \"0620f0f8-aca1-4855-bf27-239431093a89\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") "
Aug 04 19:04:05.697150 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.696992    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-certs\") pod \"0620f0f8-aca1-4855-bf27-239431093a89\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") "
Aug 04 19:04:05.697150 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-data\") pod \"0620f0f8-aca1-4855-bf27-239431093a89\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") "
Aug 04 19:04:05.697150 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697042    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-socket\") pod \"0620f0f8-aca1-4855-bf27-239431093a89\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") "
Aug 04 19:04:05.697150 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697079    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-credential-socket\") pod \"0620f0f8-aca1-4855-bf27-239431093a89\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") "
Aug 04 19:04:05.697150 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0620f0f8-aca1-4855-bf27-239431093a89-istio-podinfo\") pod \"0620f0f8-aca1-4855-bf27-239431093a89\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") "
Aug 04 19:04:05.697150 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-envoy\") pod \"0620f0f8-aca1-4855-bf27-239431093a89\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") "
Aug 04 19:04:05.697486 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697183    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdkf2\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-kube-api-access-xdkf2\") pod \"0620f0f8-aca1-4855-bf27-239431093a89\" (UID: \"0620f0f8-aca1-4855-bf27-239431093a89\") "
Aug 04 19:04:05.697486 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697372    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0620f0f8-aca1-4855-bf27-239431093a89" (UID: "0620f0f8-aca1-4855-bf27-239431093a89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:04:05.697486 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697395    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0620f0f8-aca1-4855-bf27-239431093a89" (UID: "0620f0f8-aca1-4855-bf27-239431093a89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:04:05.697674 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697483    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0620f0f8-aca1-4855-bf27-239431093a89" (UID: "0620f0f8-aca1-4855-bf27-239431093a89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:04:05.697737 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697533    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-data" (OuterVolumeSpecName: "istio-data") pod "0620f0f8-aca1-4855-bf27-239431093a89" (UID: "0620f0f8-aca1-4855-bf27-239431093a89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:04:05.698159 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.697972    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0620f0f8-aca1-4855-bf27-239431093a89-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0620f0f8-aca1-4855-bf27-239431093a89" (UID: "0620f0f8-aca1-4855-bf27-239431093a89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 19:04:05.698578 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.698554    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:04:05.698713 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.698586    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:04:05.699589 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.699561    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-istio-token" (OuterVolumeSpecName: "istio-token") pod "0620f0f8-aca1-4855-bf27-239431093a89" (UID: "0620f0f8-aca1-4855-bf27-239431093a89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:04:05.700229 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.700207    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-kube-api-access-xdkf2" (OuterVolumeSpecName: "kube-api-access-xdkf2") pod "0620f0f8-aca1-4855-bf27-239431093a89" (UID: "0620f0f8-aca1-4855-bf27-239431093a89"). InnerVolumeSpecName "kube-api-access-xdkf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:04:05.700301 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.700263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0620f0f8-aca1-4855-bf27-239431093a89-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0620f0f8-aca1-4855-bf27-239431093a89" (UID: "0620f0f8-aca1-4855-bf27-239431093a89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 04 19:04:05.700630 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.700593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0620f0f8-aca1-4855-bf27-239431093a89" (UID: "0620f0f8-aca1-4855-bf27-239431093a89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:04:05.799980 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.799941    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdkf2\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-kube-api-access-xdkf2\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:04:05.799980 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.799975    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0620f0f8-aca1-4855-bf27-239431093a89-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:04:05.799980 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.799984    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0620f0f8-aca1-4855-bf27-239431093a89-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:04:05.799980 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.799992    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:04:05.800244 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.800002    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:04:05.800244 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.800010    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0620f0f8-aca1-4855-bf27-239431093a89-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:04:05.800244 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:05.800018    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0620f0f8-aca1-4855-bf27-239431093a89-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\""
Aug 04 19:04:06.551537 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:06.551499    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"
Aug 04 19:04:06.570535 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:06.570506    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"]
Aug 04 19:04:06.576455 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:06.576428    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-2nnlh"]
Aug 04 19:04:08.545398 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:04:08.545363    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0620f0f8-aca1-4855-bf27-239431093a89" path="/var/lib/kubelet/pods/0620f0f8-aca1-4855-bf27-239431093a89/volumes"
Aug 04 19:07:09.850280 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:09.850248    2581 ???:1] "http: TLS handshake error from 10.0.135.124:42572: EOF"
Aug 04 19:07:09.851700 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:09.851680    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/0.log"
Aug 04 19:07:09.881480 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:09.881441    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/1.log"
Aug 04 19:07:10.757846 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:10.757813    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/0.log"
Aug 04 19:07:10.785939 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:10.785909    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/1.log"
Aug 04 19:07:12.071207 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:12.071176    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-v4675_c3ad50db-2758-4dc7-a5a3-a9d9994d64ab/limitador/0.log"
Aug 04 19:07:16.164932 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.164849    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m6lzs/must-gather-8kqgf"]
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165125    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="llm-d-routing-sidecar"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165137    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="llm-d-routing-sidecar"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165148    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165154    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165163    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165168    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165174    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="storage-initializer"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165179    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="storage-initializer"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165187    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="storage-initializer"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165192    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="storage-initializer"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165200    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165206    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165212    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165217    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165261    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165267    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="llm-d-routing-sidecar"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165273    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9998b74-f323-4f23-a047-cb55c566e16d" containerName="main"
Aug 04 19:07:16.165282 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165281    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0436453c-3113-4ea4-8ce8-0ddedefd115f" containerName="main"
Aug 04 19:07:16.165843 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.165365    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0620f0f8-aca1-4855-bf27-239431093a89" containerName="istio-proxy"
Aug 04 19:07:16.168063 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.168046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6lzs/must-gather-8kqgf"
Aug 04 19:07:16.170676 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.170653    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6lzs\"/\"kube-root-ca.crt\""
Aug 04 19:07:16.170799 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.170707    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6lzs\"/\"openshift-service-ca.crt\""
Aug 04 19:07:16.177801 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.177779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6lzs/must-gather-8kqgf"]
Aug 04 19:07:16.289486 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.289455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/924c8d41-296d-4820-81ee-e2f3cbb4de8b-must-gather-output\") pod \"must-gather-8kqgf\" (UID: \"924c8d41-296d-4820-81ee-e2f3cbb4de8b\") " pod="openshift-must-gather-m6lzs/must-gather-8kqgf"
Aug 04 19:07:16.289698 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.289518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mg5\" (UniqueName: \"kubernetes.io/projected/924c8d41-296d-4820-81ee-e2f3cbb4de8b-kube-api-access-p2mg5\") pod \"must-gather-8kqgf\" (UID: \"924c8d41-296d-4820-81ee-e2f3cbb4de8b\") " pod="openshift-must-gather-m6lzs/must-gather-8kqgf"
Aug 04 19:07:16.389918 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.389880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/924c8d41-296d-4820-81ee-e2f3cbb4de8b-must-gather-output\") pod \"must-gather-8kqgf\" (UID: \"924c8d41-296d-4820-81ee-e2f3cbb4de8b\") " pod="openshift-must-gather-m6lzs/must-gather-8kqgf"
Aug 04 19:07:16.390109 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.389942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mg5\" (UniqueName: \"kubernetes.io/projected/924c8d41-296d-4820-81ee-e2f3cbb4de8b-kube-api-access-p2mg5\") pod \"must-gather-8kqgf\" (UID: \"924c8d41-296d-4820-81ee-e2f3cbb4de8b\") " pod="openshift-must-gather-m6lzs/must-gather-8kqgf"
Aug 04 19:07:16.390262 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.390242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/924c8d41-296d-4820-81ee-e2f3cbb4de8b-must-gather-output\") pod \"must-gather-8kqgf\" (UID: \"924c8d41-296d-4820-81ee-e2f3cbb4de8b\") " pod="openshift-must-gather-m6lzs/must-gather-8kqgf"
Aug 04 19:07:16.405094 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.405065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mg5\" (UniqueName: \"kubernetes.io/projected/924c8d41-296d-4820-81ee-e2f3cbb4de8b-kube-api-access-p2mg5\") pod \"must-gather-8kqgf\" (UID: \"924c8d41-296d-4820-81ee-e2f3cbb4de8b\") " pod="openshift-must-gather-m6lzs/must-gather-8kqgf"
Aug 04 19:07:16.477145 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.477053    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6lzs/must-gather-8kqgf"
Aug 04 19:07:16.601830 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.601806    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6lzs/must-gather-8kqgf"]
Aug 04 19:07:16.604221 ip-10-0-139-201 kubenswrapper[2581]: W0804 19:07:16.604191    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod924c8d41_296d_4820_81ee_e2f3cbb4de8b.slice/crio-e83f68a8e0496e4eb32a5edccf220dc7cddc6b97827b5562b61095c489882599 WatchSource:0}: Error finding container e83f68a8e0496e4eb32a5edccf220dc7cddc6b97827b5562b61095c489882599: Status 404 returned error can't find the container with id e83f68a8e0496e4eb32a5edccf220dc7cddc6b97827b5562b61095c489882599
Aug 04 19:07:16.605995 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:16.605975    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 19:07:17.235784 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:17.235747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6lzs/must-gather-8kqgf" event={"ID":"924c8d41-296d-4820-81ee-e2f3cbb4de8b","Type":"ContainerStarted","Data":"e83f68a8e0496e4eb32a5edccf220dc7cddc6b97827b5562b61095c489882599"}
Aug 04 19:07:18.243176 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:18.243135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6lzs/must-gather-8kqgf" event={"ID":"924c8d41-296d-4820-81ee-e2f3cbb4de8b","Type":"ContainerStarted","Data":"907a78cdfc20a7746762f05f71f9195ae76ce0967108183087d7b49256ad0f88"}
Aug 04 19:07:18.243637 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:18.243192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6lzs/must-gather-8kqgf" event={"ID":"924c8d41-296d-4820-81ee-e2f3cbb4de8b","Type":"ContainerStarted","Data":"7009e8341841d05f4422874186ea8c91153d75cf02fcc6a28f6c4296f71a4372"}
Aug 04 19:07:18.261503 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:18.259327    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m6lzs/must-gather-8kqgf" podStartSLOduration=1.3354811500000001 podStartE2EDuration="2.25930766s" totalImagesPullingTime="923.82651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:07:18.259244073 +0000 UTC m=+4046.193313972" watchObservedRunningTime="2026-08-04 19:07:18.25930766 +0000 UTC m=+4046.193377559"
Aug 04 19:07:19.072493 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:19.072451    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-glgg2_082c2f9a-47d7-4425-90dc-55ee974b6c58/global-pull-secret-syncer/0.log"
Aug 04 19:07:19.179252 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:19.179204    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9rkkz_1f50134d-2df8-49c9-93fc-1337b4b52b3b/konnectivity-agent/0.log"
Aug 04 19:07:19.285896 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:19.285856    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-201.ec2.internal_ced7042c6bb32bfc2b5d6d18b0919b8f/haproxy/0.log"
Aug 04 19:07:24.361124 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:24.361097    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-v4675_c3ad50db-2758-4dc7-a5a3-a9d9994d64ab/limitador/0.log"
Aug 04 19:07:26.382042 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:26.381998    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97t28_5aa6b949-4a04-48e7-8760-be24399a39eb/node-exporter/0.log"
Aug 04 19:07:26.468025 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:26.467966    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97t28_5aa6b949-4a04-48e7-8760-be24399a39eb/kube-rbac-proxy/0.log"
Aug 04 19:07:26.526221 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:26.526195    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97t28_5aa6b949-4a04-48e7-8760-be24399a39eb/init-textfile/0.log"
Aug 04 19:07:29.023749 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.023714    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"]
Aug 04 19:07:29.027963 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.027931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.031001 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.030769    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m6lzs\"/\"default-dockercfg-jqf5h\""
Aug 04 19:07:29.039649 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.039623    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"]
Aug 04 19:07:29.113219 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.113056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgdz\" (UniqueName: \"kubernetes.io/projected/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-kube-api-access-pcgdz\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.113219 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.113187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-proc\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.113219 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.113223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-podres\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.113527 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.113342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-lib-modules\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.113527 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.113378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-sys\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.214722 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.214677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-proc\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.214928 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.214738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-podres\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.214928 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.214797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-lib-modules\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.214928 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.214828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-sys\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.214928 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.214878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgdz\" (UniqueName: \"kubernetes.io/projected/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-kube-api-access-pcgdz\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.215309 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.215284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-proc\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.215437 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.215417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-podres\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.215524 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.215509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-lib-modules\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.215590 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.215558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-sys\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.223620 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.223578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgdz\" (UniqueName: \"kubernetes.io/projected/2ca8c15a-81e6-4da5-8ecc-e47691b62fa7-kube-api-access-pcgdz\") pod \"perf-node-gather-daemonset-wfnwp\" (UID: \"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.342294 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.341992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:29.502123 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:29.502092    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"]
Aug 04 19:07:29.509901 ip-10-0-139-201 kubenswrapper[2581]: W0804 19:07:29.509860    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2ca8c15a_81e6_4da5_8ecc_e47691b62fa7.slice/crio-b1bef8c048a2601b26ffe3b87536ae7da6ff87983fb763da7b547f5704aafd23 WatchSource:0}: Error finding container b1bef8c048a2601b26ffe3b87536ae7da6ff87983fb763da7b547f5704aafd23: Status 404 returned error can't find the container with id b1bef8c048a2601b26ffe3b87536ae7da6ff87983fb763da7b547f5704aafd23
Aug 04 19:07:30.316478 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:30.316441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp" event={"ID":"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7","Type":"ContainerStarted","Data":"0450a7cdbd2a5316e477c14e8b82f1758ab5854dbd4d860cc977ea4565184505"}
Aug 04 19:07:30.317064 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:30.317037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:30.317188 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:30.317076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp" event={"ID":"2ca8c15a-81e6-4da5-8ecc-e47691b62fa7","Type":"ContainerStarted","Data":"b1bef8c048a2601b26ffe3b87536ae7da6ff87983fb763da7b547f5704aafd23"}
Aug 04 19:07:30.336647 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:30.336578    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp" podStartSLOduration=1.336556315 podStartE2EDuration="1.336556315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:07:30.333691266 +0000 UTC m=+4058.267761169" watchObservedRunningTime="2026-08-04 19:07:30.336556315 +0000 UTC m=+4058.270626216"
Aug 04 19:07:31.400844 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:31.400805    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mks8f_3ec1403a-a186-47a0-8a5d-c7853dca9d20/dns/0.log"
Aug 04 19:07:31.423268 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:31.423239    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mks8f_3ec1403a-a186-47a0-8a5d-c7853dca9d20/kube-rbac-proxy/0.log"
Aug 04 19:07:31.536128 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:31.536098    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rl4zg_1e8bdc52-e2e6-4679-8ff8-de9ba96c3770/dns-node-resolver/0.log"
Aug 04 19:07:32.088808 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:32.088768    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-599d78bb5d-4dgsq_50848535-20e5-443f-9da0-1132f2280608/registry/0.log"
Aug 04 19:07:32.113454 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:32.113428    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mm2xh_caccac04-758f-4f30-91d8-d2c828ab88d7/node-ca/0.log"
Aug 04 19:07:33.316054 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:33.316022    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/0.log"
Aug 04 19:07:33.353505 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:33.353464    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/1.log"
Aug 04 19:07:33.944558 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:33.944531    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2nvr2_350365eb-919f-41fc-aff6-2903c573215b/serve-healthcheck-canary/0.log"
Aug 04 19:07:34.446217 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:34.446188    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nqb98_6fb0db2a-5076-4279-b371-c8af54f232b6/kube-rbac-proxy/0.log"
Aug 04 19:07:34.468878 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:34.468845    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nqb98_6fb0db2a-5076-4279-b371-c8af54f232b6/exporter/0.log"
Aug 04 19:07:34.498993 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:34.498968    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nqb98_6fb0db2a-5076-4279-b371-c8af54f232b6/extractor/0.log"
Aug 04 19:07:36.329620 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:36.329571    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-wfnwp"
Aug 04 19:07:37.632692 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:37.632657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68dff9b49f-gtlwx_3a51f473-6ceb-4ca5-97cc-3bd07f911e53/manager/0.log"
Aug 04 19:07:38.849207 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:38.849174    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/0.log"
Aug 04 19:07:38.895395 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:38.895361    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/1.log"
Aug 04 19:07:39.723905 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:39.723867    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dxpmm_a3e2c34a-24c5-40fd-ace8-ffc777105646/s3-init/0.log"
Aug 04 19:07:39.751043 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:39.750936    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dxpmm_a3e2c34a-24c5-40fd-ace8-ffc777105646/download-hf-model/0.log"
Aug 04 19:07:40.174810 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:40.174778    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/0.log"
Aug 04 19:07:40.214891 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:40.214859    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-d92tq_4b38bfa6-6ffe-4039-a812-07860e2747e4/istio-proxy/1.log"
Aug 04 19:07:44.460435 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:44.460395    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-zsqt6_8e2d2773-14f9-42c9-8d3b-a4573c0f800d/migrator/0.log"
Aug 04 19:07:44.482978 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:44.482950    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-zsqt6_8e2d2773-14f9-42c9-8d3b-a4573c0f800d/graceful-termination/0.log"
Aug 04 19:07:45.990977 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:45.990949    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh9j2_d7e66ed3-d135-4ce6-af93-b463a55a2b6f/kube-multus-additional-cni-plugins/0.log"
Aug 04 19:07:46.014807 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:46.014780    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh9j2_d7e66ed3-d135-4ce6-af93-b463a55a2b6f/egress-router-binary-copy/0.log"
Aug 04 19:07:46.043237 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:46.043195    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh9j2_d7e66ed3-d135-4ce6-af93-b463a55a2b6f/cni-plugins/0.log"
Aug 04 19:07:46.069682 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:46.069651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh9j2_d7e66ed3-d135-4ce6-af93-b463a55a2b6f/bond-cni-plugin/0.log"
Aug 04 19:07:46.094598 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:46.094567    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh9j2_d7e66ed3-d135-4ce6-af93-b463a55a2b6f/routeoverride-cni/0.log"
Aug 04 19:07:46.118335 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:46.118300    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh9j2_d7e66ed3-d135-4ce6-af93-b463a55a2b6f/whereabouts-cni-bincopy/0.log"
Aug 04 19:07:46.144061 ip-10-0-139-201 kubenswrapper[2581]: I0804 19:07:46.143990    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh9j2_d7e66ed3-d135-4ce6-af93-b463a55a2b6f/whereabouts-cni/0.log"