Sep 16 18:39:44.357217 ip-10-0-143-44 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:39:44.774074 ip-10-0-143-44 kubenswrapper[2590]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:44.774074 ip-10-0-143-44 kubenswrapper[2590]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:39:44.774074 ip-10-0-143-44 kubenswrapper[2590]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:44.774074 ip-10-0-143-44 kubenswrapper[2590]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:39:44.774074 ip-10-0-143-44 kubenswrapper[2590]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:44.774753 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.774083    2590 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:39:44.779966 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779942    2590 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:44.779966 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779962    2590 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:44.779966 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779966    2590 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:44.779966 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779968    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:44.779966 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779972    2590 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:44.779966 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779975    2590 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779979    2590 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779982    2590 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779985    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779988    2590 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779990    2590 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779993    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779995    2590 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.779998    2590 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780000    2590 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780003    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780006    2590 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780008    2590 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780011    2590 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780013    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780016    2590 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780018    2590 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780021    2590 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780023    2590 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780026    2590 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:44.780194 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780028    2590 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780034    2590 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780036    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780039    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780041    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780044    2590 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780046    2590 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780048    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780051    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780053    2590 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780055    2590 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780060    2590 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780064    2590 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780067    2590 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780071    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780074    2590 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780077    2590 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780080    2590 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780082    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:44.780678 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780085    2590 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780088    2590 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780090    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780093    2590 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780096    2590 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780099    2590 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780101    2590 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780104    2590 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780106    2590 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780109    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780111    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780114    2590 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780117    2590 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780119    2590 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780122    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780124    2590 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780128    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780130    2590 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780132    2590 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:44.781200 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780135    2590 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780138    2590 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780141    2590 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780143    2590 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780146    2590 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780148    2590 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780151    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780154    2590 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780156    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780159    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780162    2590 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780165    2590 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780168    2590 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780170    2590 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780173    2590 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780175    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780178    2590 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780181    2590 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780183    2590 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780186    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:44.781661 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780189    2590 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780192    2590 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780194    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780197    2590 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780200    2590 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780202    2590 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780206    2590 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780209    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780212    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780214    2590 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780216    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780219    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780222    2590 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780224    2590 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780227    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780231    2590 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780972    2590 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780982    2590 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780985    2590 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:44.782170 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780988    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780992    2590 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780995    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.780998    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781000    2590 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781003    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781005    2590 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781008    2590 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781010    2590 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781013    2590 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781015    2590 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781018    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781021    2590 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781024    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781027    2590 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781030    2590 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781032    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781035    2590 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781037    2590 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781040    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:44.782631 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781043    2590 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781046    2590 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781048    2590 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781051    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781053    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781056    2590 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781058    2590 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781061    2590 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781064    2590 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781066    2590 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781069    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781072    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781074    2590 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781077    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781079    2590 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781082    2590 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781085    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781087    2590 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781089    2590 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781092    2590 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:44.783141 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781094    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781097    2590 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781099    2590 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781102    2590 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781104    2590 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781107    2590 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781109    2590 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781113    2590 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781115    2590 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781118    2590 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781120    2590 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781125    2590 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781129    2590 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781134    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781137    2590 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781140    2590 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781143    2590 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781146    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781148    2590 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:44.783621 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781151    2590 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781154    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781156    2590 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781159    2590 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781162    2590 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781165    2590 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781168    2590 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781171    2590 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781174    2590 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781176    2590 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781179    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781182    2590 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781184    2590 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781187    2590 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781189    2590 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781192    2590 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781194    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781197    2590 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781199    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781202    2590 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:44.784098 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781204    2590 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781209    2590 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781212    2590 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781214    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781218    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781220    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781223    2590 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781225    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781229    2590 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781232    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781235    2590 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781237    2590 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781240    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781242    2590 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781245    2590 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781248    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781250    2590 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781662    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781667    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:44.784595 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781671    2590 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781674    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781677    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781680    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781683    2590 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781686    2590 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781688    2590 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781691    2590 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781694    2590 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781696    2590 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781700    2590 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781704    2590 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781707    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781710    2590 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781714    2590 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781717    2590 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781720    2590 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781722    2590 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781725    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:44.785086 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781728    2590 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781731    2590 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781733    2590 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781736    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781738    2590 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781741    2590 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781743    2590 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781745    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781748    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781751    2590 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781755    2590 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781757    2590 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781759    2590 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781762    2590 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781765    2590 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781767    2590 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781769    2590 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781772    2590 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781775    2590 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781777    2590 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:44.785542 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781780    2590 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781782    2590 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781784    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781787    2590 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781789    2590 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781791    2590 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781794    2590 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781800    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781803    2590 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781806    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781808    2590 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781811    2590 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781813    2590 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781816    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781818    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781821    2590 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781823    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781826    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781828    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781830    2590 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:44.786045 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781833    2590 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781835    2590 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781839    2590 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781842    2590 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781844    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781846    2590 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781850    2590 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781853    2590 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781856    2590 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781859    2590 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781861    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781864    2590 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781867    2590 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781869    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781871    2590 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781874    2590 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781892    2590 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781895    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781899    2590 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:44.786562 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781904    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781906    2590 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781909    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781912    2590 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781914    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781917    2590 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781919    2590 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781922    2590 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781924    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781927    2590 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781929    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781932    2590 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781934    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781937    2590 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781939    2590 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781941    2590 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781944    2590 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781946    2590 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.781949    2590 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782018    2590 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:39:44.787042 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782053    2590 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782060    2590 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782064    2590 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782068    2590 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782071    2590 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782075    2590 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782079    2590 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782082    2590 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782085    2590 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782088    2590 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782091    2590 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782096    2590 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782100    2590 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782103    2590 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782106    2590 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782110    2590 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782113    2590 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782117    2590 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782120    2590 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782123    2590 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782126    2590 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782129    2590 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782132    2590 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:39:44.787520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782135    2590 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782138    2590 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782141    2590 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782145    2590 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782147    2590 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782150    2590 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782153    2590 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782156    2590 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782160    2590 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782163    2590 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782165    2590 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782168    2590 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782171    2590 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782174    2590 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782179    2590 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782182    2590 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782185    2590 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782188    2590 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782191    2590 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782194    2590 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782198    2590 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782202    2590 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782205    2590 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782208    2590 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782211    2590 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:39:44.788151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782215    2590 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782217    2590 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782220    2590 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782223    2590 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782226    2590 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782229    2590 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782232    2590 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782235    2590 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782238    2590 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782241    2590 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782244    2590 flags.go:64] FLAG: --help="false"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782247    2590 flags.go:64] FLAG: --hostname-override="ip-10-0-143-44.ec2.internal"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782250    2590 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782253    2590 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782256    2590 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782259    2590 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782262    2590 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782265    2590 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782267    2590 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782270    2590 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782273    2590 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782276    2590 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782278    2590 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782281    2590 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:39:44.788773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782284    2590 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782286    2590 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782289    2590 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782293    2590 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782297    2590 flags.go:64] FLAG: --lock-file=""
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782300    2590 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782303    2590 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782306    2590 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782311    2590 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782315    2590 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782317    2590 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782320    2590 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782323    2590 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782326    2590 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782329    2590 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782332    2590 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782336    2590 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782339    2590 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782343    2590 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782346    2590 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782349    2590 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782352    2590 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782354    2590 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782357    2590 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782360    2590 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:39:44.789361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782363    2590 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782370    2590 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782373    2590 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782376    2590 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782379    2590 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782382    2590 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782387    2590 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782390    2590 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782393    2590 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782396    2590 flags.go:64] FLAG: --port="10250"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782399    2590 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782402    2590 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d7142fe951ad4fa2"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782407    2590 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782410    2590 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782413    2590 flags.go:64] FLAG: --register-node="true"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782415    2590 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782419    2590 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782422    2590 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782425    2590 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782428    2590 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782431    2590 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782434    2590 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782437    2590 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782440    2590 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782443    2590 flags.go:64] FLAG: --runonce="false"
Sep 16 18:39:44.789987 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782445    2590 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782448    2590 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782451    2590 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782454    2590 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782457    2590 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782460    2590 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782463    2590 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782466    2590 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782468    2590 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782471    2590 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782473    2590 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782477    2590 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782480    2590 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782483    2590 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782485    2590 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782491    2590 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782493    2590 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782496    2590 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782500    2590 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782504    2590 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782507    2590 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782510    2590 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782512    2590 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782515    2590 flags.go:64] FLAG: --v="2"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782519    2590 flags.go:64] FLAG: --version="false"
Sep 16 18:39:44.790597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782524    2590 flags.go:64] FLAG: --vmodule=""
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782528    2590 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782531    2590 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782646    2590 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782650    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782653    2590 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782656    2590 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782659    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782663    2590 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782665    2590 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782668    2590 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782671    2590 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782674    2590 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782677    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782679    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782682    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782684    2590 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782687    2590 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782689    2590 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782692    2590 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:44.791244 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782694    2590 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782697    2590 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782700    2590 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782703    2590 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782705    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782709    2590 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782713    2590 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782716    2590 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782719    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782721    2590 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782724    2590 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782726    2590 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782729    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782731    2590 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782734    2590 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782737    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782739    2590 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782741    2590 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782744    2590 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782746    2590 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:44.791828 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782749    2590 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782751    2590 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782754    2590 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782756    2590 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782758    2590 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782761    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782763    2590 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782765    2590 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782768    2590 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782770    2590 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782773    2590 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782775    2590 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782778    2590 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782780    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782784    2590 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782787    2590 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782790    2590 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782794    2590 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782797    2590 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782800    2590 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:44.792329 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782803    2590 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782805    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782808    2590 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782811    2590 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782813    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782816    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782820    2590 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782823    2590 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782825    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782828    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782830    2590 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782833    2590 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782836    2590 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782838    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782840    2590 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782843    2590 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782846    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782850    2590 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782853    2590 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:44.792825 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782855    2590 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782858    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782861    2590 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782863    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782866    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782869    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782872    2590 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782892    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782898    2590 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782902    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782906    2590 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782910    2590 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782913    2590 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782916    2590 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782919    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782921    2590 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782924    2590 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782927    2590 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782930    2590 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782932    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:44.793318 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782935    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782938    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.782940    2590 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.782949    2590 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.789996    2590 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.790013    2590 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.790770    2590 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.790781    2590 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790837    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790841    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790844    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790848    2590 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790851    2590 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790854    2590 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790857    2590 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:44.793811 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790860    2590 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790863    2590 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790866    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790869    2590 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790872    2590 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790874    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790897    2590 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790900    2590 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790903    2590 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790906    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790908    2590 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790911    2590 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790914    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790916    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790919    2590 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790922    2590 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790924    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790927    2590 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790929    2590 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790932    2590 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:44.794205 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790934    2590 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790937    2590 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790942    2590 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790945    2590 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790947    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790950    2590 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790952    2590 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790955    2590 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790958    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790960    2590 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790963    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790965    2590 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790967    2590 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790970    2590 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790972    2590 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790975    2590 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790977    2590 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790980    2590 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790983    2590 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:44.794714 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790987    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790990    2590 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790993    2590 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790995    2590 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.790999    2590 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791003    2590 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791006    2590 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791008    2590 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791012    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791014    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791017    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791020    2590 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791022    2590 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791024    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791027    2590 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791030    2590 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791033    2590 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791036    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791038    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791041    2590 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:44.795195 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791044    2590 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791046    2590 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791048    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791052    2590 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791056    2590 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791058    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791061    2590 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791064    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791067    2590 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791070    2590 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791073    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791075    2590 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791078    2590 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791081    2590 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791084    2590 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791086    2590 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791088    2590 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791091    2590 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791094    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:44.795681 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791096    2590 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791098    2590 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791101    2590 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791104    2590 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791107    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791109    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791112    2590 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791115    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791118    2590 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791120    2590 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791124    2590 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791126    2590 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791129    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791132    2590 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.791137    2590 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:44.796196 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791250    2590 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791255    2590 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791258    2590 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791260    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791263    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791266    2590 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791269    2590 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791271    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791274    2590 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791276    2590 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791278    2590 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791281    2590 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791283    2590 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791286    2590 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791288    2590 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791291    2590 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791293    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791296    2590 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791298    2590 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791300    2590 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:44.796580 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791303    2590 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791305    2590 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791308    2590 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791311    2590 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791313    2590 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791316    2590 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791319    2590 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791322    2590 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791325    2590 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791327    2590 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791330    2590 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791332    2590 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791335    2590 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791337    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791340    2590 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791343    2590 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791346    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791349    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791351    2590 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:44.797076 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791355    2590 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791358    2590 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791361    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791364    2590 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791367    2590 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791369    2590 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791372    2590 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791375    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791377    2590 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791379    2590 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791382    2590 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791384    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791386    2590 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791389    2590 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791391    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791394    2590 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791396    2590 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791399    2590 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791402    2590 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:44.797528 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791404    2590 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791407    2590 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791409    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791412    2590 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791415    2590 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791417    2590 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791419    2590 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791422    2590 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791424    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791427    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791429    2590 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791431    2590 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791434    2590 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791436    2590 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791439    2590 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791441    2590 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791445    2590 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791448    2590 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791451    2590 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791454    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:44.797999 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791457    2590 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791459    2590 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791462    2590 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791464    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791467    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791470    2590 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791472    2590 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791474    2590 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791477    2590 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791480    2590 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791482    2590 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791485    2590 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791487    2590 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791489    2590 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791492    2590 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791494    2590 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791497    2590 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791499    2590 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791502    2590 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791504    2590 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:44.798502 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:44.791507    2590 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:44.799315 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.791511    2590 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:44.799315 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.791623    2590 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:39:44.799315 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.793605    2590 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:39:44.799315 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.794594    2590 server.go:1019] "Starting client certificate rotation"
Sep 16 18:39:44.799315 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.794687    2590 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:44.799315 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.794716    2590 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:44.799315 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.797808    2590 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:44.802600 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.802573    2590 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:44.816462 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.816438    2590 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:39:44.820109 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.820092    2590 log.go:25] "Validated CRI v1 image API"
Sep 16 18:39:44.821113 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.821097    2590 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:39:44.826210 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.826185    2590 fs.go:135] Filesystem UUIDs: map[33babdd6-b4e1-4175-966d-c0d47285f460:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a8cde2e7-b385-4045-bf79-e4e4ef6e768a:/dev/nvme0n1p3]
Sep 16 18:39:44.826307 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.826210    2590 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:39:44.830272 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.830254    2590 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:44.831480 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.831373    2590 manager.go:216] Machine: {Timestamp:2026-09-16 18:39:44.830005591 +0000 UTC m=+0.354464416 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100314 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dd11e0e9e77393c89f68991a1772b SystemUUID:ec2dd11e-0e9e-7739-3c89-f68991a1772b BootID:a80d7ef2-b851-48ee-9e3c-0bb4c168406b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:a4:2f:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:a4:2f:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:a1:66:a2:50:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:39:44.831480 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.831476    2590 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:39:44.831591 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.831559    2590 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:39:44.834201 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.834173    2590 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:39:44.834356 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.834204    2590 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-44.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:39:44.835051 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.835040    2590 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:39:44.835090 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.835053    2590 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:39:44.835090 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.835065    2590 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:44.835817 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.835808    2590 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:44.836574 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.836564    2590 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:39:44.840119 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.840108    2590 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:44.840230 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.840221    2590 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:39:44.840260 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.840250    2590 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:39:44.840287 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.840260    2590 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:39:44.840287 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.840275    2590 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:39:44.840287 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.840285    2590 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:39:44.840375 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.840293    2590 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:39:44.840690 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.840679    2590 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:44.840736 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.840720    2590 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:44.843232 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.843216    2590 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:39:44.846238 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.846219    2590 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:39:44.846312 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.846253    2590 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:39:44.847839 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847824    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:39:44.847917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847844    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:39:44.847917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847851    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:39:44.847917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847859    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:39:44.847917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847871    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:39:44.847917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847895    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:39:44.847917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847910    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:39:44.847917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847915    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:39:44.847917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847922    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:39:44.848164 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847928    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:39:44.848164 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847936    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:39:44.848164 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.847941    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:39:44.849037 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.849023    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:39:44.849037 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.849035    2590 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:39:44.852718 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.852706    2590 server.go:1268] "Started kubelet"
Sep 16 18:39:44.852810 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.852782    2590 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:39:44.852870 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.852795    2590 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:39:44.852936 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.852868    2590 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:39:44.853644 ip-10-0-143-44 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:39:44.854182 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.854157    2590 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:39:44.855000 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.854968    2590 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:39:44.858645 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.858621    2590 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-44.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:39:44.859263 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.859241    2590 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:39:44.859449 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.859432    2590 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:39:44.859999 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.859977    2590 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:39:44.860768 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.860752    2590 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:44.861391 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.861364    2590 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:39:44.861499 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.861462    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:44.861939 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.861920    2590 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:39:44.861939 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.861920    2590 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:39:44.862069 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.861948    2590 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:39:44.862069 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862023    2590 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:39:44.862069 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862027    2590 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:39:44.862272 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862256    2590 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:39:44.862272 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862274    2590 factory.go:55] Registering systemd factory
Sep 16 18:39:44.862365 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862282    2590 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:39:44.862568 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862555    2590 factory.go:153] Registering CRI-O factory
Sep 16 18:39:44.862568 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862570    2590 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:39:44.862636 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862583    2590 factory.go:103] Registering Raw factory
Sep 16 18:39:44.862636 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862592    2590 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:39:44.862974 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.862956    2590 manager.go:349] Starting recovery of all containers
Sep 16 18:39:44.864673 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.864647    2590 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:39:44.864673 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.864657    2590 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:39:44.865554 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.864743    2590 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-44.ec2.internal.18d5e12710df3b68  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-44.ec2.internal,UID:ip-10-0-143-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-44.ec2.internal,},FirstTimestamp:2026-09-16 18:39:44.852679528 +0000 UTC m=+0.377138352,LastTimestamp:2026-09-16 18:39:44.852679528 +0000 UTC m=+0.377138352,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-44.ec2.internal,}"
Sep 16 18:39:44.874362 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.874189    2590 manager.go:354] Recovery completed
Sep 16 18:39:44.875608 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.875586    2590 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 18:39:44.878662 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.878647    2590 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:44.880425 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.880394    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:44.880521 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.880437    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:44.880521 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.880449    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:44.881040 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.881027    2590 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:39:44.881040 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.881039    2590 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:39:44.881161 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.881040    2590 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:44.881161 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.881053    2590 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:44.881723 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.881707    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:44.881781 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.881737    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:44.881781 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.881752    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:44.882468 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.882407    2590 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-44.ec2.internal.18d5e12712868edf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-44.ec2.internal,UID:ip-10-0-143-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-44.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-44.ec2.internal,},FirstTimestamp:2026-09-16 18:39:44.880422623 +0000 UTC m=+0.404881448,LastTimestamp:2026-09-16 18:39:44.880422623 +0000 UTC m=+0.404881448,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-44.ec2.internal,}"
Sep 16 18:39:44.883261 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.883249    2590 policy_none.go:49] "None policy: Start"
Sep 16 18:39:44.883312 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.883265    2590 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:39:44.883312 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.883275    2590 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:39:44.885238 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.885226    2590 policy_none.go:47] "Start"
Sep 16 18:39:44.915450 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.915350    2590 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-44.ec2.internal.18d5e1271286db02  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-44.ec2.internal,UID:ip-10-0-143-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-44.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-44.ec2.internal,},FirstTimestamp:2026-09-16 18:39:44.880442114 +0000 UTC m=+0.404900939,LastTimestamp:2026-09-16 18:39:44.880442114 +0000 UTC m=+0.404900939,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-44.ec2.internal,}"
Sep 16 18:39:44.920685 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.920669    2590 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:39:44.920760 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.920709    2590 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:39:44.920760 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.920721    2590 server.go:87] "Starting device plugin registration server"
Sep 16 18:39:44.921149 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.921131    2590 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.921149    2590 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.921284    2590 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.921371    2590 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.921381    2590 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.921861    2590 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.921941    2590 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.926412    2590 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-44.ec2.internal.18d5e12712870dfd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-44.ec2.internal,UID:ip-10-0-143-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-44.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-44.ec2.internal,},FirstTimestamp:2026-09-16 18:39:44.880455165 +0000 UTC m=+0.404913990,LastTimestamp:2026-09-16 18:39:44.880455165 +0000 UTC m=+0.404913990,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-44.ec2.internal,}"
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.928139    2590 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-99rr9"
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:44.934820    2590 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-99rr9"
Sep 16 18:39:44.947953 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:44.936004    2590 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-44.ec2.internal.18d5e12712868edf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-44.ec2.internal.18d5e12712868edf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-44.ec2.internal,UID:ip-10-0-143-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-44.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-44.ec2.internal,},FirstTimestamp:2026-09-16 18:39:44.880422623 +0000 UTC m=+0.404881448,LastTimestamp:2026-09-16 18:39:44.881722153 +0000 UTC m=+0.406180984,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-44.ec2.internal,}"
Sep 16 18:39:45.018039 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.017992    2590 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:39:45.019433 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.019415    2590 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:39:45.019519 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.019484    2590 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:39:45.019519 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.019514    2590 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:39:45.019615 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.019568    2590 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:39:45.021289 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.021268    2590 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:45.022191 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.022170    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:45.022285 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.022200    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:45.022285 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.022209    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:45.022285 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.022231    2590 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.022285 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.022253    2590 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:45.034421 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.034403    2590 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.034546 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.034426    2590 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-44.ec2.internal\": node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.061620 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.061587    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.120133 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.120097    2590 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal"]
Sep 16 18:39:45.120297 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.120175    2590 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:45.121644 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.121626    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:45.121750 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.121669    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:45.121750 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.121680    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:45.123993 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.123980    2590 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:45.124131 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.124117    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.124168 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.124147    2590 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:45.124788 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.124770    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:45.124896 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.124795    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:45.124896 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.124805    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:45.124896 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.124778    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:45.124896 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.124895    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:45.125086 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.124913    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:45.127088 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.127070    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.127182 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.127101    2590 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:45.127838 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.127823    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:45.127958 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.127848    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:45.127958 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.127862    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:45.146872 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.146848    2590 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-44.ec2.internal\" not found" node="ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.151420 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.151400    2590 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-44.ec2.internal\" not found" node="ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.162446 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.162428    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.163596 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.163577    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7df31d8fa2980fe3e3ed4e1097304f09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal\" (UID: \"7df31d8fa2980fe3e3ed4e1097304f09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.163646 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.163605    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/018261b2b77e73288a163ad4b4ee59f0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-44.ec2.internal\" (UID: \"018261b2b77e73288a163ad4b4ee59f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.163646 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.163622    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7df31d8fa2980fe3e3ed4e1097304f09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal\" (UID: \"7df31d8fa2980fe3e3ed4e1097304f09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.263248 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.263216    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.264352 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.264333    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7df31d8fa2980fe3e3ed4e1097304f09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal\" (UID: \"7df31d8fa2980fe3e3ed4e1097304f09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.264439 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.264365    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7df31d8fa2980fe3e3ed4e1097304f09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal\" (UID: \"7df31d8fa2980fe3e3ed4e1097304f09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.264439 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.264390    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/018261b2b77e73288a163ad4b4ee59f0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-44.ec2.internal\" (UID: \"018261b2b77e73288a163ad4b4ee59f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.264439 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.264429    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/018261b2b77e73288a163ad4b4ee59f0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-44.ec2.internal\" (UID: \"018261b2b77e73288a163ad4b4ee59f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.264439 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.264430    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7df31d8fa2980fe3e3ed4e1097304f09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal\" (UID: \"7df31d8fa2980fe3e3ed4e1097304f09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.264593 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.264431    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7df31d8fa2980fe3e3ed4e1097304f09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal\" (UID: \"7df31d8fa2980fe3e3ed4e1097304f09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.364014 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.363938    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.449214 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.449172    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.454218 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.454202    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:45.464014 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.463995    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.564478 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.564442    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.664950 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.664874    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.765330 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.765299    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.794550 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.794528    2590 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:39:45.794985 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.794667    2590 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:39:45.865544 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.865516    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:45.939286 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.939196    2590 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:34:45 +0000 UTC" deadline="2028-04-10 21:16:23.522236603 +0000 UTC"
Sep 16 18:39:45.939286 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:45.939248    2590 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13730h36m37.582994872s"
Sep 16 18:39:45.966581 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:45.966518    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:46.023484 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:46.023440    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018261b2b77e73288a163ad4b4ee59f0.slice/crio-75641378c174665b1f4ea28280b11b304c7adc3d35b6ae30f510187ce0019f34 WatchSource:0}: Error finding container 75641378c174665b1f4ea28280b11b304c7adc3d35b6ae30f510187ce0019f34: Status 404 returned error can't find the container with id 75641378c174665b1f4ea28280b11b304c7adc3d35b6ae30f510187ce0019f34
Sep 16 18:39:46.024009 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:39:46.023989    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df31d8fa2980fe3e3ed4e1097304f09.slice/crio-0c0adc0f4b12b98b1cb5a65841b4fa1a160fa5c9106c7488b3e35341d1d33540 WatchSource:0}: Error finding container 0c0adc0f4b12b98b1cb5a65841b4fa1a160fa5c9106c7488b3e35341d1d33540: Status 404 returned error can't find the container with id 0c0adc0f4b12b98b1cb5a65841b4fa1a160fa5c9106c7488b3e35341d1d33540
Sep 16 18:39:46.030437 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.030420    2590 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:39:46.066776 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.066734    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:46.075563 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.075544    2590 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:46.076729 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.076711    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:46.076837 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.076742    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:46.076837 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.076755    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:46.167958 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.167915    2590 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-44.ec2.internal\" not found"
Sep 16 18:39:46.177555 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.177530    2590 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:46.268233 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.268132    2590 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:46.281324 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.281301    2590 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:46.283698 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.283681    2590 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal"
Sep 16 18:39:46.301240 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.301220    2590 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:46.408667 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.408639    2590 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:46.426351 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.426319    2590 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:46.841239 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.841204    2590 apiserver.go:52] "Watching apiserver"
Sep 16 18:39:46.849960 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.849926    2590 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:39:46.851204 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.851176    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-9gwts","openshift-multus/multus-bvhwm","openshift-network-operator/iptables-alerter-847cd","openshift-ovn-kubernetes/ovnkube-node-vrd5m","kube-system/global-pull-secret-syncer-qdskf","kube-system/konnectivity-agent-kn9bc","openshift-cluster-node-tuning-operator/tuned-s7c7w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal","openshift-multus/multus-additional-cni-plugins-swzz6","openshift-multus/network-metrics-daemon-pg5zg","openshift-network-diagnostics/network-check-target-682pg","kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq","openshift-dns/node-resolver-nwx8j"]
Sep 16 18:39:46.853975 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.853948    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:46.854086 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.854051    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:39:46.856351 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.856322    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:46.856465 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.856408    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:39:46.859796 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.859775    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.861255 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.861235    2590 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:46.861875 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.861851    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.862762 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.862739    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:46.862859 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.862805    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:46.862934 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.862859    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-prqfq\""
Sep 16 18:39:46.863016 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.862993    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:39:46.864278 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.864259    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.865513 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.865489    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:39:46.865513 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.865504    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:39:46.865679 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.865577    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:39:46.865679 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.865608    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:39:46.865823 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.865687    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:39:46.865823 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.865695    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6rjpw\""
Sep 16 18:39:46.866353 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.866322    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:39:46.866593 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.866574    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:39:46.867983 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.867916    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:39:46.870632 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870611    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:46.870691 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870641    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:39:46.870691 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870660    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-run-openvswitch\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.870757 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870689    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-log-socket\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.870757 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870724    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ca209f9-abbd-4384-9a5f-4afe95f095ac-ovnkube-script-lib\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.870827 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870760    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-systemd-units\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.870827 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870793    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c752cc3d-aa4e-477a-8347-8e2820392135-hosts-file\") pod \"node-resolver-nwx8j\" (UID: \"c752cc3d-aa4e-477a-8347-8e2820392135\") " pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.870889 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870824    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2228aeeb-cc9b-4bb8-98b3-d8eda32dff73-iptables-alerter-script\") pod \"iptables-alerter-847cd\" (UID: \"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73\") " pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.870889 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870851    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-kubelet\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.870973 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870890    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-run-ovn\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.870973 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870917    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-cni-netd\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871045 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.870972    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871045 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871008    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ca209f9-abbd-4384-9a5f-4afe95f095ac-ovn-node-metrics-cert\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871045 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871036    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dttq\" (UniqueName: \"kubernetes.io/projected/5ca209f9-abbd-4384-9a5f-4afe95f095ac-kube-api-access-6dttq\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871138 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871072    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972lx\" (UniqueName: \"kubernetes.io/projected/5989f247-505b-4069-92dc-8d563bfeaae6-kube-api-access-972lx\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:46.871138 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871104    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-node-log\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871202 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871159    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:46.871236 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871222    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c752cc3d-aa4e-477a-8347-8e2820392135-tmp-dir\") pod \"node-resolver-nwx8j\" (UID: \"c752cc3d-aa4e-477a-8347-8e2820392135\") " pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.871270 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871259    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4c6\" (UniqueName: \"kubernetes.io/projected/2228aeeb-cc9b-4bb8-98b3-d8eda32dff73-kube-api-access-rw4c6\") pod \"iptables-alerter-847cd\" (UID: \"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73\") " pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.871306 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871285    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-run-systemd\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871341 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871314    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-etc-openvswitch\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871341 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871324    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:39:46.871434 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871415    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871470 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871455    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-slash\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871506 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871486    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-run-netns\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871584 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871527    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-cni-bin\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871619 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871602    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ca209f9-abbd-4384-9a5f-4afe95f095ac-env-overrides\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871655 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871636    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58x7d\" (UniqueName: \"kubernetes.io/projected/c752cc3d-aa4e-477a-8347-8e2820392135-kube-api-access-58x7d\") pod \"node-resolver-nwx8j\" (UID: \"c752cc3d-aa4e-477a-8347-8e2820392135\") " pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.871698 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871667    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:46.871748 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871698    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2228aeeb-cc9b-4bb8-98b3-d8eda32dff73-host-slash\") pod \"iptables-alerter-847cd\" (UID: \"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73\") " pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.871806 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871728    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ca209f9-abbd-4384-9a5f-4afe95f095ac-ovnkube-config\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871860 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871818    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-var-lib-openvswitch\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.871910 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.871901    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:39:46.872354 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.872333    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f94h2\""
Sep 16 18:39:46.872354 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.872348    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5nvj7\""
Sep 16 18:39:46.873591 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.873565    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:46.876237 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.876215    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.877179 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.877160    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:39:46.877430 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.877409    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:39:46.877814 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.877728    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f2jbf\""
Sep 16 18:39:46.878220 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.878088    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6jwkz\""
Sep 16 18:39:46.878220 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.878105    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:39:46.878220 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.878087    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:39:46.878220 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.878164    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:39:46.878478 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.878457    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:39:46.879226 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.879207    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:46.879652 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.879634    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:39:46.880833 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.880308    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:39:46.880833 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.880691    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:39:46.881050 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.881030    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:39:46.882507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.881787    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p4cc5\""
Sep 16 18:39:46.882507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.881801    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.882507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.882252    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-whsh9\""
Sep 16 18:39:46.882507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.882282    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:39:46.882507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.882391    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:39:46.882507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.882453    2590 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:46.884088 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.884070    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jtwtb\""
Sep 16 18:39:46.884983 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.884436    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:46.884983 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.884505    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:39:46.884983 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.884754    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:46.885169 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.885010    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:46.916388 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.916359    2590 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kq4hd"
Sep 16 18:39:46.928622 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.928287    2590 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kq4hd"
Sep 16 18:39:46.963328 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.963302    2590 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:39:46.972214 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972187    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-cni-dir\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.972387 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972218    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-modprobe-d\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.972387 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972254    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-sysctl-d\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.972387 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972273    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-sysctl-conf\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.972387 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972295    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-etc-openvswitch\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.972387 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972333    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/191f585d-d90c-4b3c-8792-8a0d4f6abb19-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:46.972387 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972361    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-slash\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972389    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-run-netns\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972414    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-socket-dir-parent\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972436    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-etc-kubernetes\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972465    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-host\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972489    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/796bc1bc-020c-43c9-86ee-ba72e89d5b11-agent-certs\") pod \"konnectivity-agent-kn9bc\" (UID: \"796bc1bc-020c-43c9-86ee-ba72e89d5b11\") " pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972519    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58x7d\" (UniqueName: \"kubernetes.io/projected/c752cc3d-aa4e-477a-8347-8e2820392135-kube-api-access-58x7d\") pod \"node-resolver-nwx8j\" (UID: \"c752cc3d-aa4e-477a-8347-8e2820392135\") " pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972547    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-var-lib-cni-bin\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972572    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191f585d-d90c-4b3c-8792-8a0d4f6abb19-system-cni-dir\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972591    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/191f585d-d90c-4b3c-8792-8a0d4f6abb19-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972615    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-kubernetes\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972632    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-lib-modules\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972652    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-var-lib-openvswitch\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972668    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-device-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972689    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-cnibin\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972710    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-conf-dir\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972739    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/191f585d-d90c-4b3c-8792-8a0d4f6abb19-cnibin\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:46.972765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972769    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/191f585d-d90c-4b3c-8792-8a0d4f6abb19-cni-binary-copy\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972788    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcbf023b-195c-409b-b240-55139f258cb5-dbus\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972819    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-run-openvswitch\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972838    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-log-socket\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972860    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ca209f9-abbd-4384-9a5f-4afe95f095ac-ovnkube-script-lib\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972898    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqrm\" (UniqueName: \"kubernetes.io/projected/996fae34-bb65-4ab2-825b-0cc3c416329c-kube-api-access-khqrm\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972969    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-system-cni-dir\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.972990    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-var-lib-kubelet\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973019    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-var-lib-kubelet\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973044    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-run-multus-certs\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973154    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-log-socket\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973153    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcbf023b-195c-409b-b240-55139f258cb5-kubelet-config\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973200    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c752cc3d-aa4e-477a-8347-8e2820392135-hosts-file\") pod \"node-resolver-nwx8j\" (UID: \"c752cc3d-aa4e-477a-8347-8e2820392135\") " pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973214    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-etc-openvswitch\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973232    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2228aeeb-cc9b-4bb8-98b3-d8eda32dff73-iptables-alerter-script\") pod \"iptables-alerter-847cd\" (UID: \"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73\") " pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973272    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-slash\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973363    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-kubelet\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.973659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973422    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c752cc3d-aa4e-477a-8347-8e2820392135-hosts-file\") pod \"node-resolver-nwx8j\" (UID: \"c752cc3d-aa4e-477a-8347-8e2820392135\") " pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.974348 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974024    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-run-netns\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974348 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.973274    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-kubelet\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974348 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974027    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-var-lib-openvswitch\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974348 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974056    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ca209f9-abbd-4384-9a5f-4afe95f095ac-ovnkube-script-lib\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974348 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974072    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-run-openvswitch\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974540 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974522    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-run-ovn\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974587 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974559    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-cni-netd\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974587 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974567    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-run-ovn\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974676 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974620    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a91166ef-fb72-4152-87d9-7f6b9d7a127b-cni-binary-copy\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.974676 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974650    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-cni-netd\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974676 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974653    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9rrq\" (UniqueName: \"kubernetes.io/projected/191f585d-d90c-4b3c-8792-8a0d4f6abb19-kube-api-access-x9rrq\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:46.974836 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974687    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972lx\" (UniqueName: \"kubernetes.io/projected/5989f247-505b-4069-92dc-8d563bfeaae6-kube-api-access-972lx\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:46.974836 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974715    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-run-netns\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.974836 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974737    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/191f585d-d90c-4b3c-8792-8a0d4f6abb19-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:46.974836 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974737    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2228aeeb-cc9b-4bb8-98b3-d8eda32dff73-iptables-alerter-script\") pod \"iptables-alerter-847cd\" (UID: \"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73\") " pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.974836 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974755    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c752cc3d-aa4e-477a-8347-8e2820392135-tmp-dir\") pod \"node-resolver-nwx8j\" (UID: \"c752cc3d-aa4e-477a-8347-8e2820392135\") " pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.974836 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974773    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4c6\" (UniqueName: \"kubernetes.io/projected/2228aeeb-cc9b-4bb8-98b3-d8eda32dff73-kube-api-access-rw4c6\") pod \"iptables-alerter-847cd\" (UID: \"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73\") " pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.974836 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974796    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-run-systemd\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.974836 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974813    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974840    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974869    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-run\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974863    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-run-systemd\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974845    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974927    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-tmp\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974945    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-cni-bin\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974975    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ca209f9-abbd-4384-9a5f-4afe95f095ac-env-overrides\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.974999    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975023    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-var-lib-cni-multus\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975034    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-cni-bin\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975059    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-daemon-config\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975094    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/796bc1bc-020c-43c9-86ee-ba72e89d5b11-konnectivity-ca\") pod \"konnectivity-agent-kn9bc\" (UID: \"796bc1bc-020c-43c9-86ee-ba72e89d5b11\") " pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975145    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:46.975206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975174    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2228aeeb-cc9b-4bb8-98b3-d8eda32dff73-host-slash\") pod \"iptables-alerter-847cd\" (UID: \"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73\") " pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975201    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ca209f9-abbd-4384-9a5f-4afe95f095ac-ovnkube-config\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975247    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-socket-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975271    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c752cc3d-aa4e-477a-8347-8e2820392135-tmp-dir\") pod \"node-resolver-nwx8j\" (UID: \"c752cc3d-aa4e-477a-8347-8e2820392135\") " pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.975282    2590 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975271    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-sysconfig\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975327    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-tuned\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975373    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5xcd\" (UniqueName: \"kubernetes.io/projected/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-kube-api-access-z5xcd\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975418    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-sys-fs\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975431    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2228aeeb-cc9b-4bb8-98b3-d8eda32dff73-host-slash\") pod \"iptables-alerter-847cd\" (UID: \"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73\") " pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975445    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cqhn\" (UniqueName: \"kubernetes.io/projected/a91166ef-fb72-4152-87d9-7f6b9d7a127b-kube-api-access-7cqhn\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975496    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975543    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975572    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-run-k8s-cni-cncf-io\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975575    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ca209f9-abbd-4384-9a5f-4afe95f095ac-env-overrides\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.975621    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs podName:5989f247-505b-4069-92dc-8d563bfeaae6 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:47.4755917 +0000 UTC m=+3.000050512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs") pod "network-metrics-daemon-pg5zg" (UID: "5989f247-505b-4069-92dc-8d563bfeaae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:46.975770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975679    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5fb\" (UniqueName: \"kubernetes.io/projected/c12ef66b-21e1-431d-abde-fc6f7987622a-kube-api-access-9z5fb\") pod \"node-ca-9gwts\" (UID: \"c12ef66b-21e1-431d-abde-fc6f7987622a\") " pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975859    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-systemd-units\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975861    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ca209f9-abbd-4384-9a5f-4afe95f095ac-ovnkube-config\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975946    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-systemd-units\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.975990    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-hostroot\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976017    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-sys\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976051    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976103    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976126    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ca209f9-abbd-4384-9a5f-4afe95f095ac-ovn-node-metrics-cert\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976148    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dttq\" (UniqueName: \"kubernetes.io/projected/5ca209f9-abbd-4384-9a5f-4afe95f095ac-kube-api-access-6dttq\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976173    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-systemd\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976195    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c12ef66b-21e1-431d-abde-fc6f7987622a-host\") pod \"node-ca-9gwts\" (UID: \"c12ef66b-21e1-431d-abde-fc6f7987622a\") " pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976220    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c12ef66b-21e1-431d-abde-fc6f7987622a-serviceca\") pod \"node-ca-9gwts\" (UID: \"c12ef66b-21e1-431d-abde-fc6f7987622a\") " pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976259    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-node-log\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976285    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-registration-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:46.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976364    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ca209f9-abbd-4384-9a5f-4afe95f095ac-node-log\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.977053 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.976536    2590 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:39:46.980661 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.980634    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ca209f9-abbd-4384-9a5f-4afe95f095ac-ovn-node-metrics-cert\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:46.987377 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.987350    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:46.987516 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.987385    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:46.987516 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.987414    2590 projected.go:196] Error preparing data for projected volume kube-api-access-nx8vc for pod openshift-network-diagnostics/network-check-target-682pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:46.987516 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:46.987478    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc podName:167e6fbc-ccf5-4f27-b70a-616fd4a3629e nodeName:}" failed. No retries permitted until 2026-09-16 18:39:47.487457862 +0000 UTC m=+3.011916694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nx8vc" (UniqueName: "kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc") pod "network-check-target-682pg" (UID: "167e6fbc-ccf5-4f27-b70a-616fd4a3629e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:46.988127 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.988030    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58x7d\" (UniqueName: \"kubernetes.io/projected/c752cc3d-aa4e-477a-8347-8e2820392135-kube-api-access-58x7d\") pod \"node-resolver-nwx8j\" (UID: \"c752cc3d-aa4e-477a-8347-8e2820392135\") " pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:46.988127 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.988016    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4c6\" (UniqueName: \"kubernetes.io/projected/2228aeeb-cc9b-4bb8-98b3-d8eda32dff73-kube-api-access-rw4c6\") pod \"iptables-alerter-847cd\" (UID: \"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73\") " pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:46.990037 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.989571    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-972lx\" (UniqueName: \"kubernetes.io/projected/5989f247-505b-4069-92dc-8d563bfeaae6-kube-api-access-972lx\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:46.990037 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:46.989703    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dttq\" (UniqueName: \"kubernetes.io/projected/5ca209f9-abbd-4384-9a5f-4afe95f095ac-kube-api-access-6dttq\") pod \"ovnkube-node-vrd5m\" (UID: \"5ca209f9-abbd-4384-9a5f-4afe95f095ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:47.025721 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.025658    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal" event={"ID":"7df31d8fa2980fe3e3ed4e1097304f09","Type":"ContainerStarted","Data":"0c0adc0f4b12b98b1cb5a65841b4fa1a160fa5c9106c7488b3e35341d1d33540"}
Sep 16 18:39:47.027214 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.027175    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal" event={"ID":"018261b2b77e73288a163ad4b4ee59f0","Type":"ContainerStarted","Data":"75641378c174665b1f4ea28280b11b304c7adc3d35b6ae30f510187ce0019f34"}
Sep 16 18:39:47.077034 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.076992    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-run-k8s-cni-cncf-io\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.077034 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077042    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5fb\" (UniqueName: \"kubernetes.io/projected/c12ef66b-21e1-431d-abde-fc6f7987622a-kube-api-access-9z5fb\") pod \"node-ca-9gwts\" (UID: \"c12ef66b-21e1-431d-abde-fc6f7987622a\") " pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:47.077255 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077071    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-hostroot\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.077255 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077130    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-hostroot\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.077255 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077132    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-run-k8s-cni-cncf-io\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.077421 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077246    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-sys\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.077421 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077291    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-systemd\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.077421 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077332    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c12ef66b-21e1-431d-abde-fc6f7987622a-host\") pod \"node-ca-9gwts\" (UID: \"c12ef66b-21e1-431d-abde-fc6f7987622a\") " pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:47.077421 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077371    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-sys\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.077421 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077374    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c12ef66b-21e1-431d-abde-fc6f7987622a-serviceca\") pod \"node-ca-9gwts\" (UID: \"c12ef66b-21e1-431d-abde-fc6f7987622a\") " pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077435    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-systemd\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077441    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c12ef66b-21e1-431d-abde-fc6f7987622a-host\") pod \"node-ca-9gwts\" (UID: \"c12ef66b-21e1-431d-abde-fc6f7987622a\") " pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077469    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-registration-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077504    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-cni-dir\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077530    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-modprobe-d\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077558    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-sysctl-d\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077565    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-registration-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077584    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-sysctl-conf\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077616    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/191f585d-d90c-4b3c-8792-8a0d4f6abb19-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077645    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-socket-dir-parent\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077584    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-cni-dir\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077671    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-etc-kubernetes\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077673    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-modprobe-d\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.077673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077673    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-sysctl-d\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077695    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-host\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077720    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/796bc1bc-020c-43c9-86ee-ba72e89d5b11-agent-certs\") pod \"konnectivity-agent-kn9bc\" (UID: \"796bc1bc-020c-43c9-86ee-ba72e89d5b11\") " pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077746    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-var-lib-cni-bin\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077756    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-sysctl-conf\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077764    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-socket-dir-parent\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077772    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191f585d-d90c-4b3c-8792-8a0d4f6abb19-system-cni-dir\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077795    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/191f585d-d90c-4b3c-8792-8a0d4f6abb19-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077803    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-host\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077831    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-var-lib-cni-bin\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077722    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-etc-kubernetes\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077861    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191f585d-d90c-4b3c-8792-8a0d4f6abb19-system-cni-dir\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077871    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c12ef66b-21e1-431d-abde-fc6f7987622a-serviceca\") pod \"node-ca-9gwts\" (UID: \"c12ef66b-21e1-431d-abde-fc6f7987622a\") " pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077880    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-kubernetes\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077954    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/191f585d-d90c-4b3c-8792-8a0d4f6abb19-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077929    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-lib-modules\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077954    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-kubernetes\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.078392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.077997    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-device-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078054    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-device-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078083    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-cnibin\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078114    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-conf-dir\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078133    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-cnibin\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078144    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/191f585d-d90c-4b3c-8792-8a0d4f6abb19-cnibin\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078167    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-lib-modules\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078177    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-conf-dir\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078171    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/191f585d-d90c-4b3c-8792-8a0d4f6abb19-cni-binary-copy\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078219    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcbf023b-195c-409b-b240-55139f258cb5-dbus\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078232    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/191f585d-d90c-4b3c-8792-8a0d4f6abb19-cnibin\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078254    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqrm\" (UniqueName: \"kubernetes.io/projected/996fae34-bb65-4ab2-825b-0cc3c416329c-kube-api-access-khqrm\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078257    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/191f585d-d90c-4b3c-8792-8a0d4f6abb19-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078280    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-system-cni-dir\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078293    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcbf023b-195c-409b-b240-55139f258cb5-dbus\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078299    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-var-lib-kubelet\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078325    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-var-lib-kubelet\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078342    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-run-multus-certs\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.079314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078343    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-system-cni-dir\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078358    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcbf023b-195c-409b-b240-55139f258cb5-kubelet-config\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078381    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a91166ef-fb72-4152-87d9-7f6b9d7a127b-cni-binary-copy\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078378    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-var-lib-kubelet\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078386    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-var-lib-kubelet\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078401    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rrq\" (UniqueName: \"kubernetes.io/projected/191f585d-d90c-4b3c-8792-8a0d4f6abb19-kube-api-access-x9rrq\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078427    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-run-netns\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078437    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcbf023b-195c-409b-b240-55139f258cb5-kubelet-config\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078443    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/191f585d-d90c-4b3c-8792-8a0d4f6abb19-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078478    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078504    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-run\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078539    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-tmp\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078670    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/191f585d-d90c-4b3c-8792-8a0d4f6abb19-cni-binary-copy\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078775    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078791    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078810    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-var-lib-cni-multus\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078835    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-run-multus-certs\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.080284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078841    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-daemon-config\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078867    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/796bc1bc-020c-43c9-86ee-ba72e89d5b11-konnectivity-ca\") pod \"konnectivity-agent-kn9bc\" (UID: \"796bc1bc-020c-43c9-86ee-ba72e89d5b11\") " pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078878    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-run-netns\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078928    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-run\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078929    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-socket-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078842    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/191f585d-d90c-4b3c-8792-8a0d4f6abb19-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078972    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-sysconfig\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078993    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a91166ef-fb72-4152-87d9-7f6b9d7a127b-cni-binary-copy\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.078998    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-tuned\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079044    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5xcd\" (UniqueName: \"kubernetes.io/projected/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-kube-api-access-z5xcd\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079041    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079072    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-sys-fs\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079098    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cqhn\" (UniqueName: \"kubernetes.io/projected/a91166ef-fb72-4152-87d9-7f6b9d7a127b-kube-api-access-7cqhn\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079125    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.079228    2590 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.079285    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret podName:dcbf023b-195c-409b-b240-55139f258cb5 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:47.579267499 +0000 UTC m=+3.103726329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret") pod "global-pull-secret-syncer-qdskf" (UID: "dcbf023b-195c-409b-b240-55139f258cb5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079310    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a91166ef-fb72-4152-87d9-7f6b9d7a127b-multus-daemon-config\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.081378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079317    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-sys-fs\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.082302 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079382    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a91166ef-fb72-4152-87d9-7f6b9d7a127b-host-var-lib-cni-multus\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.082302 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079490    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-sysconfig\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.082302 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079493    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/996fae34-bb65-4ab2-825b-0cc3c416329c-socket-dir\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.082302 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.079930    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/796bc1bc-020c-43c9-86ee-ba72e89d5b11-konnectivity-ca\") pod \"konnectivity-agent-kn9bc\" (UID: \"796bc1bc-020c-43c9-86ee-ba72e89d5b11\") " pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:39:47.082302 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.081469    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/796bc1bc-020c-43c9-86ee-ba72e89d5b11-agent-certs\") pod \"konnectivity-agent-kn9bc\" (UID: \"796bc1bc-020c-43c9-86ee-ba72e89d5b11\") " pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:39:47.082497 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.082472    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-tmp\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.083362 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.083335    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-etc-tuned\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.087886 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.087859    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqrm\" (UniqueName: \"kubernetes.io/projected/996fae34-bb65-4ab2-825b-0cc3c416329c-kube-api-access-khqrm\") pod \"aws-ebs-csi-driver-node-pwqvq\" (UID: \"996fae34-bb65-4ab2-825b-0cc3c416329c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.088154 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.088127    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5fb\" (UniqueName: \"kubernetes.io/projected/c12ef66b-21e1-431d-abde-fc6f7987622a-kube-api-access-9z5fb\") pod \"node-ca-9gwts\" (UID: \"c12ef66b-21e1-431d-abde-fc6f7987622a\") " pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:47.088262 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.088244    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rrq\" (UniqueName: \"kubernetes.io/projected/191f585d-d90c-4b3c-8792-8a0d4f6abb19-kube-api-access-x9rrq\") pod \"multus-additional-cni-plugins-swzz6\" (UID: \"191f585d-d90c-4b3c-8792-8a0d4f6abb19\") " pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.089623 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.089604    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cqhn\" (UniqueName: \"kubernetes.io/projected/a91166ef-fb72-4152-87d9-7f6b9d7a127b-kube-api-access-7cqhn\") pod \"multus-bvhwm\" (UID: \"a91166ef-fb72-4152-87d9-7f6b9d7a127b\") " pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.090193 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.090172    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5xcd\" (UniqueName: \"kubernetes.io/projected/be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb-kube-api-access-z5xcd\") pod \"tuned-s7c7w\" (UID: \"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.176526 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.176445    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-847cd"
Sep 16 18:39:47.187174 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.187152    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:39:47.193843 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.193822    2590 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:47.197147 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.197125    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nwx8j"
Sep 16 18:39:47.204077 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.204055    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:39:47.211992 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.211966    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9gwts"
Sep 16 18:39:47.221989 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.221962    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq"
Sep 16 18:39:47.229039 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.229013    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bvhwm"
Sep 16 18:39:47.236867 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.236845    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swzz6"
Sep 16 18:39:47.244903 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.244877    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s7c7w"
Sep 16 18:39:47.483498 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.483418    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:47.483656 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.483592    2590 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:47.483706 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.483669    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs podName:5989f247-505b-4069-92dc-8d563bfeaae6 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:48.483652146 +0000 UTC m=+4.008110959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs") pod "network-metrics-daemon-pg5zg" (UID: "5989f247-505b-4069-92dc-8d563bfeaae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:47.584781 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.584511    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:47.584993 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.584798    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:47.584993 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.584715    2590 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:47.584993 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.584983    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret podName:dcbf023b-195c-409b-b240-55139f258cb5 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:48.584956911 +0000 UTC m=+4.109415776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret") pod "global-pull-secret-syncer-qdskf" (UID: "dcbf023b-195c-409b-b240-55139f258cb5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:47.584993 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.584989    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:47.585172 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.585011    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:47.585172 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.585026    2590 projected.go:196] Error preparing data for projected volume kube-api-access-nx8vc for pod openshift-network-diagnostics/network-check-target-682pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:47.585172 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:47.585077    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc podName:167e6fbc-ccf5-4f27-b70a-616fd4a3629e nodeName:}" failed. No retries permitted until 2026-09-16 18:39:48.585056092 +0000 UTC m=+4.109514906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nx8vc" (UniqueName: "kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc") pod "network-check-target-682pg" (UID: "167e6fbc-ccf5-4f27-b70a-616fd4a3629e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:47.931214 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.931160    2590 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:47 +0000 UTC" deadline="2028-04-18 02:03:06.278437817 +0000 UTC"
Sep 16 18:39:47.931214 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:47.931206    2590 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13903h23m18.347235734s"
Sep 16 18:39:48.020123 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:48.020089    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:48.020295 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:48.020213    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:39:48.492713 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:48.492679    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:48.492908 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:48.492835    2590 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:48.492971 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:48.492933    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs podName:5989f247-505b-4069-92dc-8d563bfeaae6 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:50.492891278 +0000 UTC m=+6.017350108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs") pod "network-metrics-daemon-pg5zg" (UID: "5989f247-505b-4069-92dc-8d563bfeaae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:48.593246 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:48.593156    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:48.593246 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:48.593210    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:48.593459 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:48.593383    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:48.593459 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:48.593404    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:48.593459 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:48.593417    2590 projected.go:196] Error preparing data for projected volume kube-api-access-nx8vc for pod openshift-network-diagnostics/network-check-target-682pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:48.593616 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:48.593481    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc podName:167e6fbc-ccf5-4f27-b70a-616fd4a3629e nodeName:}" failed. No retries permitted until 2026-09-16 18:39:50.593460164 +0000 UTC m=+6.117918996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nx8vc" (UniqueName: "kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc") pod "network-check-target-682pg" (UID: "167e6fbc-ccf5-4f27-b70a-616fd4a3629e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:48.593943 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:48.593918    2590 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:48.594070 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:48.593995    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret podName:dcbf023b-195c-409b-b240-55139f258cb5 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:50.593979384 +0000 UTC m=+6.118438222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret") pod "global-pull-secret-syncer-qdskf" (UID: "dcbf023b-195c-409b-b240-55139f258cb5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:48.932249 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:48.932114    2590 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:47 +0000 UTC" deadline="2028-03-18 16:30:33.765319335 +0000 UTC"
Sep 16 18:39:48.932249 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:48.932174    2590 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13173h50m44.833149573s"
Sep 16 18:39:49.023874 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.023195    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:49.023874 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:49.023348    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:39:49.023874 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.023437    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:49.023874 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:49.023512    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:39:49.045620 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.045531    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bvhwm" event={"ID":"a91166ef-fb72-4152-87d9-7f6b9d7a127b","Type":"ContainerStarted","Data":"bcff5f51f883d4a264bef0aee89ffeb9e98f11aa26d1c0d31f0882dde58b9d78"}
Sep 16 18:39:49.056796 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.053183    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9gwts" event={"ID":"c12ef66b-21e1-431d-abde-fc6f7987622a","Type":"ContainerStarted","Data":"22e431c51d49b0264b33b16ddea8e1314957651b41f2a40fb39f97b3648cc231"}
Sep 16 18:39:49.069713 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.069630    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nwx8j" event={"ID":"c752cc3d-aa4e-477a-8347-8e2820392135","Type":"ContainerStarted","Data":"cce7b44e084fa680d8ca6f6591ea72e29686c9d97c9b5ed0236f20c19c85d99b"}
Sep 16 18:39:49.080285 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.080245    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kn9bc" event={"ID":"796bc1bc-020c-43c9-86ee-ba72e89d5b11","Type":"ContainerStarted","Data":"8719939952d6b635a794049f7d9e91c396f55aa89b17a00c15e89418163288b8"}
Sep 16 18:39:49.102347 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.102300    2590 generic.go:356] "Generic (PLEG): container finished" podID="7df31d8fa2980fe3e3ed4e1097304f09" containerID="21c56ec2538a6803959a9165c42168f5c89f79ae17f51465629546324e672933" exitCode=0
Sep 16 18:39:49.102529 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.102417    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal" event={"ID":"7df31d8fa2980fe3e3ed4e1097304f09","Type":"ContainerDied","Data":"21c56ec2538a6803959a9165c42168f5c89f79ae17f51465629546324e672933"}
Sep 16 18:39:49.123928 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.123440    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal" event={"ID":"018261b2b77e73288a163ad4b4ee59f0","Type":"ContainerStarted","Data":"c62027612fe4d19a68183ece90f1dfe4f026e1e06359e41fe8ce94c89dbed92e"}
Sep 16 18:39:49.135389 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.135326    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-847cd" event={"ID":"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73","Type":"ContainerStarted","Data":"894e6d8c11811f9223ea2f6f3e9e7bf6a68c54a2de3add588ece002faa158d72"}
Sep 16 18:39:49.161149 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.160778    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" event={"ID":"5ca209f9-abbd-4384-9a5f-4afe95f095ac","Type":"ContainerStarted","Data":"ab241691e566493af18588b6bee1eeda0689918eaf5eec846d55398d54f1b717"}
Sep 16 18:39:49.178565 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.178449    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq" event={"ID":"996fae34-bb65-4ab2-825b-0cc3c416329c","Type":"ContainerStarted","Data":"57f6ce1a828208cce4a6039908ae11b5791b964469288e527a078b0f6fc6a008"}
Sep 16 18:39:49.197173 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.196982    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s7c7w" event={"ID":"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb","Type":"ContainerStarted","Data":"f33f69b917602c4b35ee5be474bd8cff442a91e5bac388c696ad08a45c3ae210"}
Sep 16 18:39:49.210098 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:49.210057    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swzz6" event={"ID":"191f585d-d90c-4b3c-8792-8a0d4f6abb19","Type":"ContainerStarted","Data":"8b07e2a12eb08d733159ad912b95b2471e3e68b050f4dcde51a75e64bf92c62b"}
Sep 16 18:39:50.020679 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:50.020637    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:50.021343 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:50.020798    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:39:50.228278 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:50.228184    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal" event={"ID":"7df31d8fa2980fe3e3ed4e1097304f09","Type":"ContainerStarted","Data":"f376bce4e135e729f337d46066bf9dbb9e7c7a68f7476bf16de9ebca8a85a80c"}
Sep 16 18:39:50.259149 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:50.258171    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-44.ec2.internal" podStartSLOduration=3.258149222 podStartE2EDuration="3.258149222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:49.151706538 +0000 UTC m=+4.676165376" watchObservedRunningTime="2026-09-16 18:39:50.258149222 +0000 UTC m=+5.782608060"
Sep 16 18:39:50.513271 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:50.513229    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:50.513469 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:50.513413    2590 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:50.513556 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:50.513481    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs podName:5989f247-505b-4069-92dc-8d563bfeaae6 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:54.513460098 +0000 UTC m=+10.037918935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs") pod "network-metrics-daemon-pg5zg" (UID: "5989f247-505b-4069-92dc-8d563bfeaae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:50.615154 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:50.614254    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:50.615154 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:50.614304    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:50.615154 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:50.614467    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:50.615154 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:50.614488    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:50.615154 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:50.614502    2590 projected.go:196] Error preparing data for projected volume kube-api-access-nx8vc for pod openshift-network-diagnostics/network-check-target-682pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:50.615154 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:50.614564    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc podName:167e6fbc-ccf5-4f27-b70a-616fd4a3629e nodeName:}" failed. No retries permitted until 2026-09-16 18:39:54.614544259 +0000 UTC m=+10.139003078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nx8vc" (UniqueName: "kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc") pod "network-check-target-682pg" (UID: "167e6fbc-ccf5-4f27-b70a-616fd4a3629e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:50.615154 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:50.615021    2590 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:50.615154 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:50.615076    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret podName:dcbf023b-195c-409b-b240-55139f258cb5 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:54.61505922 +0000 UTC m=+10.139518041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret") pod "global-pull-secret-syncer-qdskf" (UID: "dcbf023b-195c-409b-b240-55139f258cb5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:51.022825 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:51.022157    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:51.022825 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:51.022290    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:39:51.022825 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:51.022368    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:51.022825 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:51.022443    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:39:52.020836 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:52.020795    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:52.021067 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:52.020971    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:39:53.020712 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:53.020173    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:53.020712 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:53.020319    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:39:53.020712 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:53.020380    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:53.020712 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:53.020519    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:39:54.020234 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:54.020195    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:54.020421 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:54.020361    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:39:54.548226 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:54.548186    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:54.548727 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:54.548382    2590 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:54.548727 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:54.548454    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs podName:5989f247-505b-4069-92dc-8d563bfeaae6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:02.548433166 +0000 UTC m=+18.072891985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs") pod "network-metrics-daemon-pg5zg" (UID: "5989f247-505b-4069-92dc-8d563bfeaae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:54.649323 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:54.649281    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:54.649493 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:54.649335    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:54.649545 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:54.649514    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:54.649545 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:54.649537    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:54.649610 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:54.649547    2590 projected.go:196] Error preparing data for projected volume kube-api-access-nx8vc for pod openshift-network-diagnostics/network-check-target-682pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:54.649610 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:54.649596    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc podName:167e6fbc-ccf5-4f27-b70a-616fd4a3629e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:02.649582556 +0000 UTC m=+18.174041368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nx8vc" (UniqueName: "kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc") pod "network-check-target-682pg" (UID: "167e6fbc-ccf5-4f27-b70a-616fd4a3629e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:54.649950 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:54.649914    2590 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:54.650016 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:54.649972    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret podName:dcbf023b-195c-409b-b240-55139f258cb5 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:02.649946332 +0000 UTC m=+18.174405145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret") pod "global-pull-secret-syncer-qdskf" (UID: "dcbf023b-195c-409b-b240-55139f258cb5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:55.020944 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:55.020835    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:55.021112 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:55.020981    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:39:55.021112 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:55.021058    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:55.021235 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:55.021139    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:39:56.023008 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:56.022400    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:56.023008 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:56.022544    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:39:57.020543 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:57.019833    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:57.020543 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:57.019982    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:39:57.020543 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:57.020407    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:57.020543 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:57.020504    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:39:58.020112 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:58.020058    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:39:58.020504 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:58.020234    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:39:59.020285 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:59.020246    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:39:59.020754 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:59.020367    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:39:59.020754 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:39:59.020421    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:39:59.020754 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:39:59.020515    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:00.020069 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:00.019978    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:00.020230 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:00.020113    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:01.020020 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:01.019747    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:01.020492 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:01.019795    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:01.020492 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:01.020128    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:01.020492 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:01.020256    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:02.020078 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:02.020038    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:02.020579 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:02.020163    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:02.612212 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:02.612171    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:02.612390 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:02.612354    2590 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:02.612461 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:02.612439    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs podName:5989f247-505b-4069-92dc-8d563bfeaae6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:18.612416628 +0000 UTC m=+34.136875452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs") pod "network-metrics-daemon-pg5zg" (UID: "5989f247-505b-4069-92dc-8d563bfeaae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:02.712833 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:02.712798    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:02.713025 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:02.712841    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:02.713025 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:02.712982    2590 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:02.713150 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:02.713024    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:02.713150 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:02.713044    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:02.713150 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:02.713059    2590 projected.go:196] Error preparing data for projected volume kube-api-access-nx8vc for pod openshift-network-diagnostics/network-check-target-682pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:02.713150 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:02.713047    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret podName:dcbf023b-195c-409b-b240-55139f258cb5 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:18.713031646 +0000 UTC m=+34.237490458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret") pod "global-pull-secret-syncer-qdskf" (UID: "dcbf023b-195c-409b-b240-55139f258cb5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:02.713150 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:02.713120    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc podName:167e6fbc-ccf5-4f27-b70a-616fd4a3629e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:18.71310213 +0000 UTC m=+34.237560958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nx8vc" (UniqueName: "kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc") pod "network-check-target-682pg" (UID: "167e6fbc-ccf5-4f27-b70a-616fd4a3629e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:03.020072 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:03.019956    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:03.020238 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:03.020099    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:03.020238 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:03.020165    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:03.020613 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:03.020279    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:04.020569 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:04.020527    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:04.021069 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:04.020664    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:05.020959 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:05.020920    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:05.021373 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:05.020938    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:05.021373 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:05.021029    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:05.021373 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:05.021097    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:06.020277 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:06.020238    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:06.020464 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:06.020364    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:07.022131 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.021935    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:07.022131 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:07.022074    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:07.022763 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.022189    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:07.022763 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:07.022265    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:07.273266 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.273087    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" event={"ID":"5ca209f9-abbd-4384-9a5f-4afe95f095ac","Type":"ContainerStarted","Data":"f641514b65191df69827e9e6dbbea101154bc7daa1fe3fca99622c6cb97c7dec"}
Sep 16 18:40:07.273393 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.273330    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" event={"ID":"5ca209f9-abbd-4384-9a5f-4afe95f095ac","Type":"ContainerStarted","Data":"a0c71ca2d2c2d89d9752f03af437beda93d93780ee26ca0743a4f1fbb2969252"}
Sep 16 18:40:07.273393 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.273383    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" event={"ID":"5ca209f9-abbd-4384-9a5f-4afe95f095ac","Type":"ContainerStarted","Data":"80bb12183f1fa749a50ef6df5f310d970db9daa32449e7a2efb09e27a9375aa9"}
Sep 16 18:40:07.273504 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.273430    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" event={"ID":"5ca209f9-abbd-4384-9a5f-4afe95f095ac","Type":"ContainerStarted","Data":"f3eed58a4a205d4004a78985cb821d8c931558e5b89709564b2d54c1c4495865"}
Sep 16 18:40:07.273504 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.273479    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" event={"ID":"5ca209f9-abbd-4384-9a5f-4afe95f095ac","Type":"ContainerStarted","Data":"1153b2afab7692d383cb2c1436fc18bd895312537f3a5dc92fb0bc6bba69ae24"}
Sep 16 18:40:07.274526 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.274505    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq" event={"ID":"996fae34-bb65-4ab2-825b-0cc3c416329c","Type":"ContainerStarted","Data":"db6d15bf521b9a6d7ec9bce0e98e4792d1416426d9fe21f61e9d914408f6cc10"}
Sep 16 18:40:07.275910 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.275865    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s7c7w" event={"ID":"be6fa2f0-70ff-43c2-9acd-8dcaf123b1fb","Type":"ContainerStarted","Data":"774c72f477b76206d619fce55057c3a8bf15969f3c7328972b2d7dd2ed692ad5"}
Sep 16 18:40:07.277212 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.277192    2590 generic.go:356] "Generic (PLEG): container finished" podID="191f585d-d90c-4b3c-8792-8a0d4f6abb19" containerID="af98ee71970a4f710977c253d7fa5be7c2bf1904c5326a9125aa31519aca787b" exitCode=0
Sep 16 18:40:07.277298 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.277271    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swzz6" event={"ID":"191f585d-d90c-4b3c-8792-8a0d4f6abb19","Type":"ContainerDied","Data":"af98ee71970a4f710977c253d7fa5be7c2bf1904c5326a9125aa31519aca787b"}
Sep 16 18:40:07.280105 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.279133    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bvhwm" event={"ID":"a91166ef-fb72-4152-87d9-7f6b9d7a127b","Type":"ContainerStarted","Data":"9f9ec4e4a1504b16176994bbc0a007f0aa8caf0e81f00c1463710e4cb667fb24"}
Sep 16 18:40:07.282175 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.282152    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9gwts" event={"ID":"c12ef66b-21e1-431d-abde-fc6f7987622a","Type":"ContainerStarted","Data":"4c543065f4c5407491c4dc4572ed4181cf61175d6507b24e76d0606ad0ccb9b6"}
Sep 16 18:40:07.283456 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.283434    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nwx8j" event={"ID":"c752cc3d-aa4e-477a-8347-8e2820392135","Type":"ContainerStarted","Data":"60726db3b333f35dd7cffef6e43617c52fab624a0df2d39b1dad3bcb65b7a114"}
Sep 16 18:40:07.284917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.284875    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kn9bc" event={"ID":"796bc1bc-020c-43c9-86ee-ba72e89d5b11","Type":"ContainerStarted","Data":"305c891cecfda812ca68d72bb38b832fa4d39a36864f8e88fe5a81fef865da98"}
Sep 16 18:40:07.303607 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.303567    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-44.ec2.internal" podStartSLOduration=21.303554169 podStartE2EDuration="21.303554169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:50.259110138 +0000 UTC m=+5.783568977" watchObservedRunningTime="2026-09-16 18:40:07.303554169 +0000 UTC m=+22.828013003"
Sep 16 18:40:07.320712 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.320661    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nwx8j" podStartSLOduration=3.651082282 podStartE2EDuration="22.320641124s" totalImagesPullingTime="18.669558842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:07.319906782 +0000 UTC m=+22.844365616" watchObservedRunningTime="2026-09-16 18:40:07.320641124 +0000 UTC m=+22.845099956"
Sep 16 18:40:07.362151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.362103    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bvhwm" podStartSLOduration=3.629803499 podStartE2EDuration="22.362084984s" totalImagesPullingTime="18.732281485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:07.342190913 +0000 UTC m=+22.866649749" watchObservedRunningTime="2026-09-16 18:40:07.362084984 +0000 UTC m=+22.886543823"
Sep 16 18:40:07.362535 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.362508    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s7c7w" podStartSLOduration=2.670469605 podStartE2EDuration="21.362496637s" totalImagesPullingTime="18.692027032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:07.361761126 +0000 UTC m=+22.886219954" watchObservedRunningTime="2026-09-16 18:40:07.362496637 +0000 UTC m=+22.886955470"
Sep 16 18:40:07.384848 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.384791    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kn9bc" podStartSLOduration=7.047956276 podStartE2EDuration="22.384770074s" totalImagesPullingTime="15.336813798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:07.384712444 +0000 UTC m=+22.909171279" watchObservedRunningTime="2026-09-16 18:40:07.384770074 +0000 UTC m=+22.909228909"
Sep 16 18:40:07.489579 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.489543    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:07.589336 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:07.589306    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:08.020202 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.020120    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:08.020479 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:08.020274    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:08.186511 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.186489    2590 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:40:08.290009 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.289960    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-847cd" event={"ID":"2228aeeb-cc9b-4bb8-98b3-d8eda32dff73","Type":"ContainerStarted","Data":"f654ee94ebf8ab278269fd6d033d003093f6db687ed55557164aaed16118aa82"}
Sep 16 18:40:08.293593 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.293561    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" event={"ID":"5ca209f9-abbd-4384-9a5f-4afe95f095ac","Type":"ContainerStarted","Data":"8c4865f5f7459c3007615ea9c59d1259312a79ea59b55d8bbadda52d9825e79b"}
Sep 16 18:40:08.295776 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.295718    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq" event={"ID":"996fae34-bb65-4ab2-825b-0cc3c416329c","Type":"ContainerStarted","Data":"4f93da3cd19766ed2473aad492381344a87ff5f268ca07c2ce2403e89f3be5e3"}
Sep 16 18:40:08.319715 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.319660    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9gwts" podStartSLOduration=4.6508817780000005 podStartE2EDuration="23.319630561s" totalImagesPullingTime="18.668748783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:07.402740211 +0000 UTC m=+22.927199047" watchObservedRunningTime="2026-09-16 18:40:08.319630561 +0000 UTC m=+23.844089397"
Sep 16 18:40:08.319921 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.319828    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-847cd" podStartSLOduration=4.659239867 podStartE2EDuration="23.319823049s" totalImagesPullingTime="18.660583182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:08.319820202 +0000 UTC m=+23.844279017" watchObservedRunningTime="2026-09-16 18:40:08.319823049 +0000 UTC m=+23.844281882"
Sep 16 18:40:08.487145 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.487041    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:08.586572 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.586490    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:08.659833 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.659788    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:08.935374 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.935206    2590 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:40:08.186504936Z","UUID":"fb66c7c3-56bd-461a-8b38-aa14add39475","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:40:08.936855 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.936819    2590 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:40:08.936855 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:08.936852    2590 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:40:09.021942 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:09.021875    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:09.022131 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:09.022101    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:09.022207 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:09.022155    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:09.022262 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:09.022236    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:10.020791 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:10.020620    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:10.021263 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:10.020932    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:10.306508 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:10.306468    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" event={"ID":"5ca209f9-abbd-4384-9a5f-4afe95f095ac","Type":"ContainerStarted","Data":"60f3acce31c26745ba18a5ed818962a8dda21178f429789f306a6357c4e61928"}
Sep 16 18:40:10.309029 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:10.309002    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq" event={"ID":"996fae34-bb65-4ab2-825b-0cc3c416329c","Type":"ContainerStarted","Data":"7243ab5930e68bc5856decb6ba6a5e6d9d6b465214918136f4b498274146313b"}
Sep 16 18:40:10.521854 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:10.521821    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:40:10.522479 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:10.522461    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:40:10.540983 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:10.540936    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwqvq" podStartSLOduration=4.705305726 podStartE2EDuration="25.540918465s" totalImagesPullingTime="20.835612739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:10.336104376 +0000 UTC m=+25.860563212" watchObservedRunningTime="2026-09-16 18:40:10.540918465 +0000 UTC m=+26.065377298"
Sep 16 18:40:11.020230 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:11.020198    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:11.020462 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:11.020333    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:11.020462 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:11.020398    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:11.020574 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:11.020511    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:11.312094 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:11.312063    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:40:11.312560 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:11.312394    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kn9bc"
Sep 16 18:40:11.384845 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:11.384818    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:12.020785 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:12.020616    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:12.020960 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:12.020864    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:12.320448 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:12.320417    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" event={"ID":"5ca209f9-abbd-4384-9a5f-4afe95f095ac","Type":"ContainerStarted","Data":"7956a89145d95725092d9a7625a8ab07dfd58ee03ff1c3a13343816e049a8108"}
Sep 16 18:40:12.320916 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:12.320618    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:40:12.320916 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:12.320675    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:40:12.322376 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:12.322351    2590 generic.go:356] "Generic (PLEG): container finished" podID="191f585d-d90c-4b3c-8792-8a0d4f6abb19" containerID="314f9206db22f53b8e148b7c6d40eaa9b6af28092dfe9b810da23623bf64def9" exitCode=0
Sep 16 18:40:12.322488 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:12.322434    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swzz6" event={"ID":"191f585d-d90c-4b3c-8792-8a0d4f6abb19","Type":"ContainerDied","Data":"314f9206db22f53b8e148b7c6d40eaa9b6af28092dfe9b810da23623bf64def9"}
Sep 16 18:40:12.336818 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:12.336770    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:40:12.352294 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:12.352252    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m" podStartSLOduration=8.631084664 podStartE2EDuration="27.352237972s" totalImagesPullingTime="18.721153308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:12.351722742 +0000 UTC m=+27.876181580" watchObservedRunningTime="2026-09-16 18:40:12.352237972 +0000 UTC m=+27.876696852"
Sep 16 18:40:13.020192 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.020162    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:13.020192 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.020180    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:13.020493 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:13.020324    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:13.020493 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:13.020407    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:13.325081 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.325051    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:40:13.339929 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.339902    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:40:13.667177 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.667088    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pg5zg"]
Sep 16 18:40:13.667374 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.667260    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:13.667440 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:13.667368    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:13.674986 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.674957    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdskf"]
Sep 16 18:40:13.675179 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.675077    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:13.675246 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:13.675171    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:13.677095 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.677070    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-682pg"]
Sep 16 18:40:13.677216 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:13.677183    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:13.677273 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:13.677256    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:15.020992 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:15.020779    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:15.021616 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:15.020833    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:15.021616 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:15.021120    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:15.021616 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:15.021144    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:15.335064 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:15.335030    2590 generic.go:356] "Generic (PLEG): container finished" podID="191f585d-d90c-4b3c-8792-8a0d4f6abb19" containerID="8ae965779463444c04fb7688f4acd82a2accf3acefd3c8afbe9cf687ea966920" exitCode=0
Sep 16 18:40:15.335238 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:15.335104    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swzz6" event={"ID":"191f585d-d90c-4b3c-8792-8a0d4f6abb19","Type":"ContainerDied","Data":"8ae965779463444c04fb7688f4acd82a2accf3acefd3c8afbe9cf687ea966920"}
Sep 16 18:40:16.020834 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:16.020757    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:16.020992 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:16.020915    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:17.020579 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:17.020542    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:17.021006 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:17.020649    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:17.021006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:17.020715    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:17.021006 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:17.020839    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:17.342238 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:17.342202    2590 generic.go:356] "Generic (PLEG): container finished" podID="191f585d-d90c-4b3c-8792-8a0d4f6abb19" containerID="950c1de73c94d107072439c6868d14890ba5784da4fabe921c3fbc532c7af61d" exitCode=0
Sep 16 18:40:17.342413 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:17.342248    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swzz6" event={"ID":"191f585d-d90c-4b3c-8792-8a0d4f6abb19","Type":"ContainerDied","Data":"950c1de73c94d107072439c6868d14890ba5784da4fabe921c3fbc532c7af61d"}
Sep 16 18:40:18.020592 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:18.020554    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:18.021114 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:18.020703    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:18.139138 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:18.139104    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:18.215559 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:18.215523    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:18.636615 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:18.636578    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:18.636775 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:18.636717    2590 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:18.636775 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:18.636774    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs podName:5989f247-505b-4069-92dc-8d563bfeaae6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:50.636758834 +0000 UTC m=+66.161217645 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs") pod "network-metrics-daemon-pg5zg" (UID: "5989f247-505b-4069-92dc-8d563bfeaae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:18.737217 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:18.737177    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:18.737402 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:18.737225    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:18.737402 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:18.737338    2590 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:18.737402 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:18.737380    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:18.737402 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:18.737400    2590 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:18.737607 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:18.737408    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret podName:dcbf023b-195c-409b-b240-55139f258cb5 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:50.737386778 +0000 UTC m=+66.261845597 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret") pod "global-pull-secret-syncer-qdskf" (UID: "dcbf023b-195c-409b-b240-55139f258cb5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:18.737607 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:18.737412    2590 projected.go:196] Error preparing data for projected volume kube-api-access-nx8vc for pod openshift-network-diagnostics/network-check-target-682pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:18.737607 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:18.737461    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc podName:167e6fbc-ccf5-4f27-b70a-616fd4a3629e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:50.73744615 +0000 UTC m=+66.261904965 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nx8vc" (UniqueName: "kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc") pod "network-check-target-682pg" (UID: "167e6fbc-ccf5-4f27-b70a-616fd4a3629e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:19.020359 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:19.020278    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:19.020359 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:19.020278    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:19.020581 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:19.020493    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pg5zg" podUID="5989f247-505b-4069-92dc-8d563bfeaae6"
Sep 16 18:40:19.020581 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:19.020547    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-682pg" podUID="167e6fbc-ccf5-4f27-b70a-616fd4a3629e"
Sep 16 18:40:20.020778 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.020675    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:20.021275 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:20.020807    2590 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdskf" podUID="dcbf023b-195c-409b-b240-55139f258cb5"
Sep 16 18:40:20.738867 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.738784    2590 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-44.ec2.internal" event="NodeReady"
Sep 16 18:40:20.739040 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.738948    2590 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:40:20.846415 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.846373    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cjk4p"]
Sep 16 18:40:20.851651 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.851623    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7hhn9"]
Sep 16 18:40:20.851815 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.851789    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cjk4p"
Sep 16 18:40:20.855123 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.855103    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:20.857960 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.857617    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:40:20.857960 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.857617    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:40:20.857960 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.857781    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kf94w\""
Sep 16 18:40:20.858137 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.857970    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:40:20.863978 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.863958    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:40:20.864109 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.864068    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:40:20.864229 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.864210    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vxh4z\""
Sep 16 18:40:20.880413 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.880383    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cjk4p"]
Sep 16 18:40:20.900460 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.900432    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7hhn9"]
Sep 16 18:40:20.953041 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.953006    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb9df607-c5c5-41b9-86a0-45503173194f-cert\") pod \"ingress-canary-cjk4p\" (UID: \"bb9df607-c5c5-41b9-86a0-45503173194f\") " pod="openshift-ingress-canary/ingress-canary-cjk4p"
Sep 16 18:40:20.953041 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.953042    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a09e3f9-b797-41a8-9929-4f5c73ebab63-tmp-dir\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:20.953259 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.953086    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476pv\" (UniqueName: \"kubernetes.io/projected/bb9df607-c5c5-41b9-86a0-45503173194f-kube-api-access-476pv\") pod \"ingress-canary-cjk4p\" (UID: \"bb9df607-c5c5-41b9-86a0-45503173194f\") " pod="openshift-ingress-canary/ingress-canary-cjk4p"
Sep 16 18:40:20.953259 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.953112    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:20.953259 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.953147    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a09e3f9-b797-41a8-9929-4f5c73ebab63-config-volume\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:20.953259 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:20.953179    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742cm\" (UniqueName: \"kubernetes.io/projected/5a09e3f9-b797-41a8-9929-4f5c73ebab63-kube-api-access-742cm\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:21.020090 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.020014    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:21.020249 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.020173    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:21.025388 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.025364    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:21.025931 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.025729    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcswh\""
Sep 16 18:40:21.025931 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.025732    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:21.026170 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.026152    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5zf4c\""
Sep 16 18:40:21.026258 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.026231    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:21.053656 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.053613    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476pv\" (UniqueName: \"kubernetes.io/projected/bb9df607-c5c5-41b9-86a0-45503173194f-kube-api-access-476pv\") pod \"ingress-canary-cjk4p\" (UID: \"bb9df607-c5c5-41b9-86a0-45503173194f\") " pod="openshift-ingress-canary/ingress-canary-cjk4p"
Sep 16 18:40:21.053783 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.053671    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:21.053783 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.053717    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a09e3f9-b797-41a8-9929-4f5c73ebab63-config-volume\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:21.053783 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.053755    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742cm\" (UniqueName: \"kubernetes.io/projected/5a09e3f9-b797-41a8-9929-4f5c73ebab63-kube-api-access-742cm\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:21.053920 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.053784    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb9df607-c5c5-41b9-86a0-45503173194f-cert\") pod \"ingress-canary-cjk4p\" (UID: \"bb9df607-c5c5-41b9-86a0-45503173194f\") " pod="openshift-ingress-canary/ingress-canary-cjk4p"
Sep 16 18:40:21.053920 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.053805    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a09e3f9-b797-41a8-9929-4f5c73ebab63-tmp-dir\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:21.053920 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:21.053839    2590 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:21.053920 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:21.053906    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls podName:5a09e3f9-b797-41a8-9929-4f5c73ebab63 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:21.553874453 +0000 UTC m=+37.078333264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls") pod "dns-default-7hhn9" (UID: "5a09e3f9-b797-41a8-9929-4f5c73ebab63") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:21.054200 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.054177    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a09e3f9-b797-41a8-9929-4f5c73ebab63-tmp-dir\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:21.054604 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.054584    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a09e3f9-b797-41a8-9929-4f5c73ebab63-config-volume\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:21.058914 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.058873    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb9df607-c5c5-41b9-86a0-45503173194f-cert\") pod \"ingress-canary-cjk4p\" (UID: \"bb9df607-c5c5-41b9-86a0-45503173194f\") " pod="openshift-ingress-canary/ingress-canary-cjk4p"
Sep 16 18:40:21.077671 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.077635    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-742cm\" (UniqueName: \"kubernetes.io/projected/5a09e3f9-b797-41a8-9929-4f5c73ebab63-kube-api-access-742cm\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:21.079033 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.079007    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-476pv\" (UniqueName: \"kubernetes.io/projected/bb9df607-c5c5-41b9-86a0-45503173194f-kube-api-access-476pv\") pod \"ingress-canary-cjk4p\" (UID: \"bb9df607-c5c5-41b9-86a0-45503173194f\") " pod="openshift-ingress-canary/ingress-canary-cjk4p"
Sep 16 18:40:21.164085 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.164050    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cjk4p"
Sep 16 18:40:21.355450 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.355281    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swzz6" event={"ID":"191f585d-d90c-4b3c-8792-8a0d4f6abb19","Type":"ContainerStarted","Data":"85bf31b772bca0702bb740608571aa9a6601601d3aab3aa4509f97b1d3cbb825"}
Sep 16 18:40:21.464595 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.464561    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cjk4p"]
Sep 16 18:40:21.480858 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:40:21.480826    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9df607_c5c5_41b9_86a0_45503173194f.slice/crio-56c0f61fcb33a1942a6466a860d4a5001e5ce7e428e6231053e8880ac9d6bab8 WatchSource:0}: Error finding container 56c0f61fcb33a1942a6466a860d4a5001e5ce7e428e6231053e8880ac9d6bab8: Status 404 returned error can't find the container with id 56c0f61fcb33a1942a6466a860d4a5001e5ce7e428e6231053e8880ac9d6bab8
Sep 16 18:40:21.557970 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:21.557901    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:21.558085 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:21.558065    2590 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:21.558148 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:21.558138    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls podName:5a09e3f9-b797-41a8-9929-4f5c73ebab63 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.558121848 +0000 UTC m=+38.082580675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls") pod "dns-default-7hhn9" (UID: "5a09e3f9-b797-41a8-9929-4f5c73ebab63") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:22.020628 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:22.020550    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:22.024975 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:22.024949    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:22.360998 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:22.360966    2590 generic.go:356] "Generic (PLEG): container finished" podID="191f585d-d90c-4b3c-8792-8a0d4f6abb19" containerID="85bf31b772bca0702bb740608571aa9a6601601d3aab3aa4509f97b1d3cbb825" exitCode=0
Sep 16 18:40:22.361666 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:22.361047    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swzz6" event={"ID":"191f585d-d90c-4b3c-8792-8a0d4f6abb19","Type":"ContainerDied","Data":"85bf31b772bca0702bb740608571aa9a6601601d3aab3aa4509f97b1d3cbb825"}
Sep 16 18:40:22.362516 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:22.362495    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cjk4p" event={"ID":"bb9df607-c5c5-41b9-86a0-45503173194f","Type":"ContainerStarted","Data":"56c0f61fcb33a1942a6466a860d4a5001e5ce7e428e6231053e8880ac9d6bab8"}
Sep 16 18:40:22.565761 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:22.565732    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:22.565931 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:22.565893    2590 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:22.565981 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:22.565955    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls podName:5a09e3f9-b797-41a8-9929-4f5c73ebab63 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:24.565939533 +0000 UTC m=+40.090398347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls") pod "dns-default-7hhn9" (UID: "5a09e3f9-b797-41a8-9929-4f5c73ebab63") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:22.640450 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:22.640358    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:22.742809 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:22.742775    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:22.841163 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:22.841124    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:23.369097 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:23.369067    2590 generic.go:356] "Generic (PLEG): container finished" podID="191f585d-d90c-4b3c-8792-8a0d4f6abb19" containerID="e7f78f1c5d6451775c4c76c216a799c208ed3d1a57032ea4d6e5ece954672455" exitCode=0
Sep 16 18:40:23.369574 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:23.369125    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swzz6" event={"ID":"191f585d-d90c-4b3c-8792-8a0d4f6abb19","Type":"ContainerDied","Data":"e7f78f1c5d6451775c4c76c216a799c208ed3d1a57032ea4d6e5ece954672455"}
Sep 16 18:40:24.374733 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.374697    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swzz6" event={"ID":"191f585d-d90c-4b3c-8792-8a0d4f6abb19","Type":"ContainerStarted","Data":"3c0c8b20a3d2442da1cd59c3b1d1de3f91bbd2766ccbfb4cb98477bdcbc4e39a"}
Sep 16 18:40:24.376244 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.376219    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cjk4p" event={"ID":"bb9df607-c5c5-41b9-86a0-45503173194f","Type":"ContainerStarted","Data":"d85e3722d278380f55a12c1c86fd2125e891f3a5c114502a5ce1da5543c057e7"}
Sep 16 18:40:24.409651 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.409599    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-swzz6" podStartSLOduration=8.389091024 podStartE2EDuration="38.409583813s" totalImagesPullingTime="30.020492789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:24.409489354 +0000 UTC m=+39.933948212" watchObservedRunningTime="2026-09-16 18:40:24.409583813 +0000 UTC m=+39.934042648"
Sep 16 18:40:24.436899 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.436834    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cjk4p" podStartSLOduration=2.224269239 podStartE2EDuration="4.436818672s" totalImagesPullingTime="2.212549433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:24.436651644 +0000 UTC m=+39.961110479" watchObservedRunningTime="2026-09-16 18:40:24.436818672 +0000 UTC m=+39.961277507"
Sep 16 18:40:24.540550 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.540502    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:24.581353 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.581319    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:24.581477 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:24.581441    2590 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:24.581523 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:24.581499    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls podName:5a09e3f9-b797-41a8-9929-4f5c73ebab63 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.581486083 +0000 UTC m=+44.105944895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls") pod "dns-default-7hhn9" (UID: "5a09e3f9-b797-41a8-9929-4f5c73ebab63") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:24.634755 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.634657    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:24.707485 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.707452    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:24.899626 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.899550    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:24.961956 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:24.961921    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:26.302451 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:26.302414    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:28.016009 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.015978    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"]
Sep 16 18:40:28.047663 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.047630    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"]
Sep 16 18:40:28.047832 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.047737    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.050282 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.050259    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mz766\""
Sep 16 18:40:28.050396 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.050378    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:40:28.051147 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.051131    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:40:28.052411 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.052393    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:40:28.052411 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.052408    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:40:28.127832 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.127804    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-h6hs2"]
Sep 16 18:40:28.147440 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.147406    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-h6hs2"]
Sep 16 18:40:28.147581 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.147523    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.154581 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.154562    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:40:28.155689 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.155666    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:40:28.155930 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.155914    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-s58jp\""
Sep 16 18:40:28.156188 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.156171    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:40:28.161259 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.161236    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:40:28.161980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.161963    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69d9584dd8-d72wj"]
Sep 16 18:40:28.164368 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.164352    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:40:28.175453 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.175433    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.187135 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.187114    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:40:28.187227 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.187138    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:40:28.187227 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.187161    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:40:28.187227 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.187155    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:40:28.187388 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.187366    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wtd28\""
Sep 16 18:40:28.187388 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.187373    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:40:28.187487 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.187373    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:40:28.205964 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.205938    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqfn\" (UniqueName: \"kubernetes.io/projected/32eb6be9-5836-434e-968e-623a8c1b4050-kube-api-access-6zqfn\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.206059 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.205979    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/32eb6be9-5836-434e-968e-623a8c1b4050-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.206059 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.206006    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.218690 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.218663    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69d9584dd8-d72wj"]
Sep 16 18:40:28.306823 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.306793    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqfn\" (UniqueName: \"kubernetes.io/projected/32eb6be9-5836-434e-968e-623a8c1b4050-kube-api-access-6zqfn\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.307049 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.306839    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv2z\" (UniqueName: \"kubernetes.io/projected/cf98bdea-b950-4609-8382-89fc440c8bdc-kube-api-access-2kv2z\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.307049 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.306904    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/32eb6be9-5836-434e-968e-623a8c1b4050-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.307049 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.306937    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.307049 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.306974    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb13daed-c369-49b7-9e38-bdd0636bd0ad-tmp\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.307049 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307000    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb13daed-c369-49b7-9e38-bdd0636bd0ad-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.307266 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.307067    2590 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:28.307266 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307122    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb13daed-c369-49b7-9e38-bdd0636bd0ad-snapshots\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.307266 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.307151    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls podName:32eb6be9-5836-434e-968e-623a8c1b4050 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.807131631 +0000 UTC m=+44.331590443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-6t2vh" (UID: "32eb6be9-5836-434e-968e-623a8c1b4050") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:28.307266 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307215    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb13daed-c369-49b7-9e38-bdd0636bd0ad-service-ca-bundle\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.307266 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307246    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pgqd\" (UniqueName: \"kubernetes.io/projected/eb13daed-c369-49b7-9e38-bdd0636bd0ad-kube-api-access-2pgqd\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.307561 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307271    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.307561 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307333    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb13daed-c369-49b7-9e38-bdd0636bd0ad-serving-cert\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.307561 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307376    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-stats-auth\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.307561 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307410    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-default-certificate\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.307561 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307434    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.307723 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.307706    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/32eb6be9-5836-434e-968e-623a8c1b4050-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.324088 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.324066    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqfn\" (UniqueName: \"kubernetes.io/projected/32eb6be9-5836-434e-968e-623a8c1b4050-kube-api-access-6zqfn\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.408134 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408101    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb13daed-c369-49b7-9e38-bdd0636bd0ad-snapshots\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.408134 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408132    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb13daed-c369-49b7-9e38-bdd0636bd0ad-service-ca-bundle\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.408326 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408151    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pgqd\" (UniqueName: \"kubernetes.io/projected/eb13daed-c369-49b7-9e38-bdd0636bd0ad-kube-api-access-2pgqd\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.408326 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408166    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.408326 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408191    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb13daed-c369-49b7-9e38-bdd0636bd0ad-serving-cert\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.408326 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408216    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-stats-auth\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.408326 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408243    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-default-certificate\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.408326 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408265    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.408326 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408299    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv2z\" (UniqueName: \"kubernetes.io/projected/cf98bdea-b950-4609-8382-89fc440c8bdc-kube-api-access-2kv2z\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.408326 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.408305    2590 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:40:28.408711 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408368    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb13daed-c369-49b7-9e38-bdd0636bd0ad-tmp\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.408711 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.408389    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.908366826 +0000 UTC m=+44.432825638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : secret "router-metrics-certs-default" not found
Sep 16 18:40:28.408711 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408421    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb13daed-c369-49b7-9e38-bdd0636bd0ad-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.408711 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.408577    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.908556514 +0000 UTC m=+44.433015352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : configmap references non-existent config key: service-ca.crt
Sep 16 18:40:28.408956 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408859    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb13daed-c369-49b7-9e38-bdd0636bd0ad-service-ca-bundle\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.408956 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408907    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb13daed-c369-49b7-9e38-bdd0636bd0ad-snapshots\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.408956 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.408930    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb13daed-c369-49b7-9e38-bdd0636bd0ad-tmp\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.409493 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.409473    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb13daed-c369-49b7-9e38-bdd0636bd0ad-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.410803 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.410774    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb13daed-c369-49b7-9e38-bdd0636bd0ad-serving-cert\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.410943 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.410897    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-stats-auth\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.410989 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.410960    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-default-certificate\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.424080 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.424061    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv2z\" (UniqueName: \"kubernetes.io/projected/cf98bdea-b950-4609-8382-89fc440c8bdc-kube-api-access-2kv2z\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.425683 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.425662    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pgqd\" (UniqueName: \"kubernetes.io/projected/eb13daed-c369-49b7-9e38-bdd0636bd0ad-kube-api-access-2pgqd\") pod \"insights-operator-5844b856db-h6hs2\" (UID: \"eb13daed-c369-49b7-9e38-bdd0636bd0ad\") " pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.456074 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.456046    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-h6hs2"
Sep 16 18:40:28.571336 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.571259    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:28.609466 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.609434    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:28.609558 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.609544    2590 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:28.609604 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.609595    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls podName:5a09e3f9-b797-41a8-9929-4f5c73ebab63 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.609582639 +0000 UTC m=+52.134041451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls") pod "dns-default-7hhn9" (UID: "5a09e3f9-b797-41a8-9929-4f5c73ebab63") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:28.679660 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.679630    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:28.760930 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.760898    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-h6hs2"]
Sep 16 18:40:28.763913 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:40:28.763865    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb13daed_c369_49b7_9e38_bdd0636bd0ad.slice/crio-4a2998f7c47fd1c5c3ff786f01b48b032951989d01aea710570f10e31749a26a WatchSource:0}: Error finding container 4a2998f7c47fd1c5c3ff786f01b48b032951989d01aea710570f10e31749a26a: Status 404 returned error can't find the container with id 4a2998f7c47fd1c5c3ff786f01b48b032951989d01aea710570f10e31749a26a
Sep 16 18:40:28.811283 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.811242    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:28.811448 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.811397    2590 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:28.811497 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.811463    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls podName:32eb6be9-5836-434e-968e-623a8c1b4050 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:29.811448215 +0000 UTC m=+45.335907028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-6t2vh" (UID: "32eb6be9-5836-434e-968e-623a8c1b4050") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:28.912102 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.912018    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.912102 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:28.912070    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:28.912291 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.912169    2590 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:40:28.912291 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.912206    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:29.912192416 +0000 UTC m=+45.436651228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : configmap references non-existent config key: service-ca.crt
Sep 16 18:40:28.912291 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:28.912228    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:29.912214072 +0000 UTC m=+45.436672884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : secret "router-metrics-certs-default" not found
Sep 16 18:40:29.388953 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:29.388918    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h6hs2" event={"ID":"eb13daed-c369-49b7-9e38-bdd0636bd0ad","Type":"ContainerStarted","Data":"4a2998f7c47fd1c5c3ff786f01b48b032951989d01aea710570f10e31749a26a"}
Sep 16 18:40:29.817538 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:29.817492    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:29.817838 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:29.817666    2590 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:29.817838 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:29.817734    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls podName:32eb6be9-5836-434e-968e-623a8c1b4050 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.817719758 +0000 UTC m=+47.342178571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-6t2vh" (UID: "32eb6be9-5836-434e-968e-623a8c1b4050") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:29.918080 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:29.918037    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:29.918265 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:29.918120    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:29.918265 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:29.918190    2590 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:40:29.918265 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:29.918257    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.918236913 +0000 UTC m=+47.442695736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : secret "router-metrics-certs-default" not found
Sep 16 18:40:29.918383 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:29.918282    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.918270638 +0000 UTC m=+47.442729450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : configmap references non-existent config key: service-ca.crt
Sep 16 18:40:31.395283 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.395241    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h6hs2" event={"ID":"eb13daed-c369-49b7-9e38-bdd0636bd0ad","Type":"ContainerStarted","Data":"7f87f3d254184d9337625d82e77117d65da1fae31ee6b770e8f39cdfba030143"}
Sep 16 18:40:31.417360 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.417310    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-h6hs2" podStartSLOduration=0.995139108 podStartE2EDuration="3.417289678s" totalImagesPullingTime="2.42215057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:31.417094832 +0000 UTC m=+46.941553672" watchObservedRunningTime="2026-09-16 18:40:31.417289678 +0000 UTC m=+46.941748513"
Sep 16 18:40:31.590211 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.590172    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"]
Sep 16 18:40:31.620803 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.620764    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"]
Sep 16 18:40:31.620994 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.620913    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.623969 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.623943    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:40:31.624113 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.624037    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zwvlk\""
Sep 16 18:40:31.624194 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.624125    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:40:31.624279 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.624270    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:31.624903 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.624861    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:31.733938 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.733908    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5da5b179-92bd-4308-a524-a08566a60d2d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5lrj2\" (UID: \"5da5b179-92bd-4308-a524-a08566a60d2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.734117 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.733964    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5da5b179-92bd-4308-a524-a08566a60d2d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5lrj2\" (UID: \"5da5b179-92bd-4308-a524-a08566a60d2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.734117 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.733986    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stpnd\" (UniqueName: \"kubernetes.io/projected/5da5b179-92bd-4308-a524-a08566a60d2d-kube-api-access-stpnd\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5lrj2\" (UID: \"5da5b179-92bd-4308-a524-a08566a60d2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.834627 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.834597    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stpnd\" (UniqueName: \"kubernetes.io/projected/5da5b179-92bd-4308-a524-a08566a60d2d-kube-api-access-stpnd\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5lrj2\" (UID: \"5da5b179-92bd-4308-a524-a08566a60d2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.834786 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.834646    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:31.834786 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:31.834740    2590 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:31.834853 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:31.834794    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls podName:32eb6be9-5836-434e-968e-623a8c1b4050 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:35.834779612 +0000 UTC m=+51.359238425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-6t2vh" (UID: "32eb6be9-5836-434e-968e-623a8c1b4050") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:31.834853 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.834828    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5da5b179-92bd-4308-a524-a08566a60d2d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5lrj2\" (UID: \"5da5b179-92bd-4308-a524-a08566a60d2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.834979 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.834863    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5da5b179-92bd-4308-a524-a08566a60d2d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5lrj2\" (UID: \"5da5b179-92bd-4308-a524-a08566a60d2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.835426 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.835402    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5da5b179-92bd-4308-a524-a08566a60d2d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5lrj2\" (UID: \"5da5b179-92bd-4308-a524-a08566a60d2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.837195 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.837175    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5da5b179-92bd-4308-a524-a08566a60d2d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5lrj2\" (UID: \"5da5b179-92bd-4308-a524-a08566a60d2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.844901 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.844828    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stpnd\" (UniqueName: \"kubernetes.io/projected/5da5b179-92bd-4308-a524-a08566a60d2d-kube-api-access-stpnd\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5lrj2\" (UID: \"5da5b179-92bd-4308-a524-a08566a60d2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.929633 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.929595    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"
Sep 16 18:40:31.936078 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.936043    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:31.936226 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:31.936098    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:31.936226 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:31.936191    2590 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:40:31.936332 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:31.936253    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:35.936236554 +0000 UTC m=+51.460695366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : secret "router-metrics-certs-default" not found
Sep 16 18:40:31.936332 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:31.936277    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:35.936259067 +0000 UTC m=+51.460717899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : configmap references non-existent config key: service-ca.crt
Sep 16 18:40:32.201501 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.201367    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:32.222810 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.222776    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2"]
Sep 16 18:40:32.226023 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:40:32.225994    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da5b179_92bd_4308_a524_a08566a60d2d.slice/crio-5e211420df557cb535eeda6938948c80ee9dc09d9afb97836562c8aa67532077 WatchSource:0}: Error finding container 5e211420df557cb535eeda6938948c80ee9dc09d9afb97836562c8aa67532077: Status 404 returned error can't find the container with id 5e211420df557cb535eeda6938948c80ee9dc09d9afb97836562c8aa67532077
Sep 16 18:40:32.302135 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.302099    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:32.399360 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.399327    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2" event={"ID":"5da5b179-92bd-4308-a524-a08566a60d2d","Type":"ContainerStarted","Data":"5e211420df557cb535eeda6938948c80ee9dc09d9afb97836562c8aa67532077"}
Sep 16 18:40:32.579528 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.579496    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:32.598912 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.598865    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tq2g6"]
Sep 16 18:40:32.633076 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.633047    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tq2g6"]
Sep 16 18:40:32.633229 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.633165    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.638112 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.638089    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:40:32.638256 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.638090    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:40:32.638256 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.638189    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-txfhn\""
Sep 16 18:40:32.663523 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.663470    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:32.742019 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.741979    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73588ab8-b492-432b-953e-68467d462260-crio-socket\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.742019 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.742020    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.742226 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.742038    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8b9v\" (UniqueName: \"kubernetes.io/projected/73588ab8-b492-432b-953e-68467d462260-kube-api-access-b8b9v\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.742226 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.742118    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73588ab8-b492-432b-953e-68467d462260-data-volume\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.742226 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.742144    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.843225 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.843133    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73588ab8-b492-432b-953e-68467d462260-data-volume\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.843225 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.843175    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.843449 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.843322    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73588ab8-b492-432b-953e-68467d462260-crio-socket\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.843449 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.843347    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.843449 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.843370    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8b9v\" (UniqueName: \"kubernetes.io/projected/73588ab8-b492-432b-953e-68467d462260-kube-api-access-b8b9v\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.843449 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:32.843377    2590 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:32.843449 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:32.843439    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.34342496 +0000 UTC m=+48.867883772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:32.843683 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.843453    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73588ab8-b492-432b-953e-68467d462260-data-volume\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.843683 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:32.843471    2590 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:32.843683 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:32.843540    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.34351364 +0000 UTC m=+48.867972469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:32.843683 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.843569    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73588ab8-b492-432b-953e-68467d462260-crio-socket\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:32.858245 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:32.858211    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8b9v\" (UniqueName: \"kubernetes.io/projected/73588ab8-b492-432b-953e-68467d462260-kube-api-access-b8b9v\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:33.311353 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.311308    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"]
Sep 16 18:40:33.332807 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.332765    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"]
Sep 16 18:40:33.333017 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.332921    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.336814 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.336613    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:40:33.337242 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.337226    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:33.337986 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.337798    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:33.337986 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.337856    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-c2c7w\""
Sep 16 18:40:33.337986 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.337871    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:40:33.346897 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.346860    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:33.347022 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.346945    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:33.347022 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:33.346961    2590 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:33.347137 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:33.347024    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.347004121 +0000 UTC m=+49.871462947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:33.347137 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:33.347054    2590 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:33.347137 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:33.347104    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.347088333 +0000 UTC m=+49.871547150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:33.448311 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.448270    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2e9887b-61ad-49be-9252-a113339cd4ff-config\") pod \"service-ca-operator-cf7c776b7-b68lv\" (UID: \"c2e9887b-61ad-49be-9252-a113339cd4ff\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.448743 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.448426    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2e9887b-61ad-49be-9252-a113339cd4ff-serving-cert\") pod \"service-ca-operator-cf7c776b7-b68lv\" (UID: \"c2e9887b-61ad-49be-9252-a113339cd4ff\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.448743 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.448466    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9frx\" (UniqueName: \"kubernetes.io/projected/c2e9887b-61ad-49be-9252-a113339cd4ff-kube-api-access-h9frx\") pod \"service-ca-operator-cf7c776b7-b68lv\" (UID: \"c2e9887b-61ad-49be-9252-a113339cd4ff\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.548860 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.548823    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9frx\" (UniqueName: \"kubernetes.io/projected/c2e9887b-61ad-49be-9252-a113339cd4ff-kube-api-access-h9frx\") pod \"service-ca-operator-cf7c776b7-b68lv\" (UID: \"c2e9887b-61ad-49be-9252-a113339cd4ff\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.549069 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.548942    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2e9887b-61ad-49be-9252-a113339cd4ff-config\") pod \"service-ca-operator-cf7c776b7-b68lv\" (UID: \"c2e9887b-61ad-49be-9252-a113339cd4ff\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.549069 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.548996    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2e9887b-61ad-49be-9252-a113339cd4ff-serving-cert\") pod \"service-ca-operator-cf7c776b7-b68lv\" (UID: \"c2e9887b-61ad-49be-9252-a113339cd4ff\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.549471 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.549451    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2e9887b-61ad-49be-9252-a113339cd4ff-config\") pod \"service-ca-operator-cf7c776b7-b68lv\" (UID: \"c2e9887b-61ad-49be-9252-a113339cd4ff\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.551593 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.551571    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2e9887b-61ad-49be-9252-a113339cd4ff-serving-cert\") pod \"service-ca-operator-cf7c776b7-b68lv\" (UID: \"c2e9887b-61ad-49be-9252-a113339cd4ff\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.559584 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.559561    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9frx\" (UniqueName: \"kubernetes.io/projected/c2e9887b-61ad-49be-9252-a113339cd4ff-kube-api-access-h9frx\") pod \"service-ca-operator-cf7c776b7-b68lv\" (UID: \"c2e9887b-61ad-49be-9252-a113339cd4ff\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.653158 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.653071    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"
Sep 16 18:40:33.949207 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:33.949125    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv"]
Sep 16 18:40:34.290764 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:34.290738    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nwx8j_c752cc3d-aa4e-477a-8347-8e2820392135/dns-node-resolver/0.log"
Sep 16 18:40:34.356747 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:34.356712    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:34.356900 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:34.356794    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:34.356900 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:34.356864    2590 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:34.356900 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:34.356866    2590 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:34.357011 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:34.356939    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.356923974 +0000 UTC m=+51.881382789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:34.357011 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:34.356954    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.356947169 +0000 UTC m=+51.881405980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:34.398049 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:40:34.398002    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e9887b_61ad_49be_9252_a113339cd4ff.slice/crio-ab85c0b04debf80a56f624f0c98ffe9dda4a629021bd79aa189467031f4f50bd WatchSource:0}: Error finding container ab85c0b04debf80a56f624f0c98ffe9dda4a629021bd79aa189467031f4f50bd: Status 404 returned error can't find the container with id ab85c0b04debf80a56f624f0c98ffe9dda4a629021bd79aa189467031f4f50bd
Sep 16 18:40:34.405362 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:34.405338    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv" event={"ID":"c2e9887b-61ad-49be-9252-a113339cd4ff","Type":"ContainerStarted","Data":"ab85c0b04debf80a56f624f0c98ffe9dda4a629021bd79aa189467031f4f50bd"}
Sep 16 18:40:35.409898 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:35.409852    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2" event={"ID":"5da5b179-92bd-4308-a524-a08566a60d2d","Type":"ContainerStarted","Data":"43a1c70c921514e43a9db423a4d34a3a5b82f0b94f91ffe3d171c5288168286b"}
Sep 16 18:40:35.431322 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:35.431273    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2" podStartSLOduration=2.025831947 podStartE2EDuration="4.431256287s" totalImagesPullingTime="2.40542434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:35.430863212 +0000 UTC m=+50.955322046" watchObservedRunningTime="2026-09-16 18:40:35.431256287 +0000 UTC m=+50.955715121"
Sep 16 18:40:35.504019 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:35.503958    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9gwts_c12ef66b-21e1-431d-abde-fc6f7987622a/node-ca/0.log"
Sep 16 18:40:35.871294 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:35.871245    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:35.871687 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:35.871377    2590 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:35.871687 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:35.871462    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls podName:32eb6be9-5836-434e-968e-623a8c1b4050 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.871440562 +0000 UTC m=+59.395899397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-6t2vh" (UID: "32eb6be9-5836-434e-968e-623a8c1b4050") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:35.972261 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:35.972223    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:35.972433 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:35.972304    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:35.972433 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:35.972365    2590 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:40:35.972433 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:35.972434    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.972419964 +0000 UTC m=+59.496878776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : secret "router-metrics-certs-default" not found
Sep 16 18:40:35.972583 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:35.972471    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.972453075 +0000 UTC m=+59.496911910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : configmap references non-existent config key: service-ca.crt
Sep 16 18:40:36.090142 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:36.090079    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cjk4p_bb9df607-c5c5-41b9-86a0-45503173194f/serve-healthcheck-canary/0.log"
Sep 16 18:40:36.376464 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:36.376418    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:36.376655 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:36.376536    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:36.376655 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:36.376583    2590 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:36.376655 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:36.376619    2590 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:36.376655 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:36.376654    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:40.376634333 +0000 UTC m=+55.901093159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:36.376858 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:36.376673    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:40.376661644 +0000 UTC m=+55.901120455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:36.619260 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:36.619232    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:36.678908 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:36.678800    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:36.679046 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:36.678951    2590 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:36.679046 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:36.679004    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls podName:5a09e3f9-b797-41a8-9929-4f5c73ebab63 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:52.678990947 +0000 UTC m=+68.203449758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls") pod "dns-default-7hhn9" (UID: "5a09e3f9-b797-41a8-9929-4f5c73ebab63") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:36.719041 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:36.719000    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:37.416546 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:37.416509    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv" event={"ID":"c2e9887b-61ad-49be-9252-a113339cd4ff","Type":"ContainerStarted","Data":"605536a8f788765c99d68e4f7a2983642b7c27ca11d44792791ed18872489d43"}
Sep 16 18:40:37.435179 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:37.435123    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv" podStartSLOduration=1.9745765020000001 podStartE2EDuration="4.435107355s" totalImagesPullingTime="2.460530853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:37.434991752 +0000 UTC m=+52.959450587" watchObservedRunningTime="2026-09-16 18:40:37.435107355 +0000 UTC m=+52.959566183"
Sep 16 18:40:37.681134 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:37.681051    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:37.746367 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:37.746325    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:38.792959 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:38.792921    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:38.880996 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:38.880958    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:39.520918 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:39.520871    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:39.614259 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:39.614222    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:40.407628 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.407582    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:40.408109 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.407687    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:40.408109 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:40.407744    2590 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:40.408109 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:40.407803    2590 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:40.408109 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:40.407809    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:48.407794668 +0000 UTC m=+63.932253480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:40.408109 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:40.407864    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:48.407850946 +0000 UTC m=+63.932309758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:40.688896 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.688757    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:40.723951 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.723920    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"]
Sep 16 18:40:40.725889 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.725863    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.732941 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.732922    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:40:40.733027 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.732922    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:40:40.733027 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.732979    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:40:40.733027 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.733019    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:40:40.733933 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.733913    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:40:40.734015 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.733947    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:40:40.734015 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.733954    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:40:40.768995 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.768967    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"]
Sep 16 18:40:40.810373 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.810340    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.810373 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.810375    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.810563 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.810409    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-hub\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.810563 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.810485    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-ca\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.810563 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.810552    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8700d590-924f-4777-9234-564844aba86b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.810673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.810576    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwhm\" (UniqueName: \"kubernetes.io/projected/8700d590-924f-4777-9234-564844aba86b-kube-api-access-snwhm\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.820088 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.820060    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:40.911841 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.911809    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8700d590-924f-4777-9234-564844aba86b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.911841 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.911845    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwhm\" (UniqueName: \"kubernetes.io/projected/8700d590-924f-4777-9234-564844aba86b-kube-api-access-snwhm\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.912110 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.911903    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.912110 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.911931    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.912110 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.911967    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-hub\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.912252 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.912135    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-ca\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.912558 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.912534    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8700d590-924f-4777-9234-564844aba86b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.914576 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.914555    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.914688 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.914667    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.914926 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.914910    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-ca\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.915060 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.915044    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8700d590-924f-4777-9234-564844aba86b-hub\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:40.921208 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:40.921186    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwhm\" (UniqueName: \"kubernetes.io/projected/8700d590-924f-4777-9234-564844aba86b-kube-api-access-snwhm\") pod \"cluster-proxy-proxy-agent-55fb4fc48c-lkvm8\" (UID: \"8700d590-924f-4777-9234-564844aba86b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:41.042027 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:41.042002    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:40:41.259494 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:41.259465    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:41.343152 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:40:41.343124    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8700d590_924f_4777_9234_564844aba86b.slice/crio-fb688c6b30668ba8b42baed47ce9141a3c0c92758b28545de86f9d30389aa3b9 WatchSource:0}: Error finding container fb688c6b30668ba8b42baed47ce9141a3c0c92758b28545de86f9d30389aa3b9: Status 404 returned error can't find the container with id fb688c6b30668ba8b42baed47ce9141a3c0c92758b28545de86f9d30389aa3b9
Sep 16 18:40:41.343152 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:41.343133    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"]
Sep 16 18:40:41.357427 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:41.357401    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:41.428030 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:41.427990    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" event={"ID":"8700d590-924f-4777-9234-564844aba86b","Type":"ContainerStarted","Data":"fb688c6b30668ba8b42baed47ce9141a3c0c92758b28545de86f9d30389aa3b9"}
Sep 16 18:40:43.767634 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:43.767599    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:43.938612 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:43.938568    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:43.938803 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:43.938762    2590 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:43.938862 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:43.938854    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls podName:32eb6be9-5836-434e-968e-623a8c1b4050 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:59.938835359 +0000 UTC m=+75.463294171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-6t2vh" (UID: "32eb6be9-5836-434e-968e-623a8c1b4050") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:40:44.039788 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:44.039741    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:44.040002 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:44.039813    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:40:44.040002 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:44.039937    2590 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:40:44.040002 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:44.039966    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:41:00.03994493 +0000 UTC m=+75.564403761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : configmap references non-existent config key: service-ca.crt
Sep 16 18:40:44.040002 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:44.039988    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs podName:cf98bdea-b950-4609-8382-89fc440c8bdc nodeName:}" failed. No retries permitted until 2026-09-16 18:41:00.039977002 +0000 UTC m=+75.564435836 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs") pod "router-default-69d9584dd8-d72wj" (UID: "cf98bdea-b950-4609-8382-89fc440c8bdc") : secret "router-metrics-certs-default" not found
Sep 16 18:40:45.348552 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:45.348526    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrd5m"
Sep 16 18:40:45.442274 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:45.442237    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" event={"ID":"8700d590-924f-4777-9234-564844aba86b","Type":"ContainerStarted","Data":"c79b7692ba8e7d61017d704aa06c5dd6785b183911f204f417639b29eb90ada4"}
Sep 16 18:40:45.442274 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:45.442277    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" event={"ID":"8700d590-924f-4777-9234-564844aba86b","Type":"ContainerStarted","Data":"e1a3a86d30d8adf178f8312fe8a0ee7cd7d171764b94867434c662183e91abad"}
Sep 16 18:40:45.442516 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:45.442290    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" event={"ID":"8700d590-924f-4777-9234-564844aba86b","Type":"ContainerStarted","Data":"9b1490dc0d494a5d55df0ab45c2d69da8dd8c917ffe1f289235c7007a280b68e"}
Sep 16 18:40:45.486988 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:45.486861    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" podStartSLOduration=1.61309113 podStartE2EDuration="5.486838388s" totalImagesPullingTime="3.873747258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:45.486600486 +0000 UTC m=+61.011059332" watchObservedRunningTime="2026-09-16 18:40:45.486838388 +0000 UTC m=+61.011297221"
Sep 16 18:40:45.751476 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:45.751386    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:46.553162 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:46.553131    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:46.770286 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:46.770254    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:46.864659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:46.864568    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:46.972796 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:46.972766    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:47.082942 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:47.082907    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:47.332114 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:47.332082    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:47.478379 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:47.478336    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:48.056932 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:48.056898    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:48.479052 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:48.478959    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:48.479052 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:48.479042    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:48.479258 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:48.479107    2590 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:48.479258 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:48.479174    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls podName:73588ab8-b492-432b-953e-68467d462260 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:04.479156965 +0000 UTC m=+80.003615781 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tq2g6" (UID: "73588ab8-b492-432b-953e-68467d462260") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:48.479492 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:48.479475    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73588ab8-b492-432b-953e-68467d462260-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:40:48.663746 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:48.663712    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:48.868134 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:48.868095    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:49.098041 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:49.098005    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:49.499692 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:49.499656    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:50.072400 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.072349    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:50.265656 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.265623    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:50.381213 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.381135    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:50.471965 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.471926    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:50.653052 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.652969    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:50.700928 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.700899    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:40:50.703378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.703355    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:50.711420 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:50.711404    2590 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:40:50.711508 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:40:50.711467    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs podName:5989f247-505b-4069-92dc-8d563bfeaae6 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:54.711451681 +0000 UTC m=+130.235910493 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs") pod "network-metrics-daemon-pg5zg" (UID: "5989f247-505b-4069-92dc-8d563bfeaae6") : secret "metrics-daemon-secret" not found
Sep 16 18:40:50.801929 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.801868    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:50.802099 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.802010    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:50.806277 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.806261    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:50.806339 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.806282    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:50.814401 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.814374    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcbf023b-195c-409b-b240-55139f258cb5-original-pull-secret\") pod \"global-pull-secret-syncer-qdskf\" (UID: \"dcbf023b-195c-409b-b240-55139f258cb5\") " pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:50.816252 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.816236    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:50.825275 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.825251    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8vc\" (UniqueName: \"kubernetes.io/projected/167e6fbc-ccf5-4f27-b70a-616fd4a3629e-kube-api-access-nx8vc\") pod \"network-check-target-682pg\" (UID: \"167e6fbc-ccf5-4f27-b70a-616fd4a3629e\") " pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:50.831052 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.831033    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdskf"
Sep 16 18:40:50.852290 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:50.852261    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.041861 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.041835    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5zf4c\""
Sep 16 18:40:51.048799 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.048778    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:51.065482 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.065455    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.154494 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.154421    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdskf"]
Sep 16 18:40:51.157378 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:40:51.157351    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbf023b_195c_409b_b240_55139f258cb5.slice/crio-c0101f965fbea40e3b9b9bf01d8ad9bae7655f727acde5b22b2cb5db012fbe03 WatchSource:0}: Error finding container c0101f965fbea40e3b9b9bf01d8ad9bae7655f727acde5b22b2cb5db012fbe03: Status 404 returned error can't find the container with id c0101f965fbea40e3b9b9bf01d8ad9bae7655f727acde5b22b2cb5db012fbe03
Sep 16 18:40:51.191965 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.191931    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.314660 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.314630    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.346640 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.346606    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-682pg"]
Sep 16 18:40:51.349965 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:40:51.349941    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167e6fbc_ccf5_4f27_b70a_616fd4a3629e.slice/crio-197b7f5306e6692c6e9a4e767031b88efd087db9a1f2fe13ac60a2729aeb0e80 WatchSource:0}: Error finding container 197b7f5306e6692c6e9a4e767031b88efd087db9a1f2fe13ac60a2729aeb0e80: Status 404 returned error can't find the container with id 197b7f5306e6692c6e9a4e767031b88efd087db9a1f2fe13ac60a2729aeb0e80
Sep 16 18:40:51.454492 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.454409    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.460604 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.460575    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-682pg" event={"ID":"167e6fbc-ccf5-4f27-b70a-616fd4a3629e","Type":"ContainerStarted","Data":"197b7f5306e6692c6e9a4e767031b88efd087db9a1f2fe13ac60a2729aeb0e80"}
Sep 16 18:40:51.461628 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:51.461607    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdskf" event={"ID":"dcbf023b-195c-409b-b240-55139f258cb5","Type":"ContainerStarted","Data":"c0101f965fbea40e3b9b9bf01d8ad9bae7655f727acde5b22b2cb5db012fbe03"}
Sep 16 18:40:52.067543 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:52.067510    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:52.446297 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:52.446192    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:52.721601 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:52.721512    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:52.724827 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:52.724800    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a09e3f9-b797-41a8-9929-4f5c73ebab63-metrics-tls\") pod \"dns-default-7hhn9\" (UID: \"5a09e3f9-b797-41a8-9929-4f5c73ebab63\") " pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:52.850385 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:52.850348    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:52.973561 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:52.973297    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vxh4z\""
Sep 16 18:40:52.981084 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:52.981050    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:53.054779 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:53.054751    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:53.236396 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:53.236308    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:53.277643 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:53.277564    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7hhn9"]
Sep 16 18:40:53.471102 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:53.471064    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7hhn9" event={"ID":"5a09e3f9-b797-41a8-9929-4f5c73ebab63","Type":"ContainerStarted","Data":"b5817590224b944c724b2ef37d27af9291c1edeacd1d536ce1226502e0915a4b"}
Sep 16 18:40:53.655665 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:53.655623    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:53.832580 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:53.832541    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:54.444501 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:54.444465    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:54.838608 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:54.838577    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:55.073819 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:55.073774    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:55.509503 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:55.509468    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:56.007898 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:56.007853    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:56.228403 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:56.228321    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:56.443193 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:56.443162    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:56.482917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:56.482825    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdskf" event={"ID":"dcbf023b-195c-409b-b240-55139f258cb5","Type":"ContainerStarted","Data":"12fb05af401652bbcc3046d1ac09970ee3046cdc1841d3b195f16836445f5381"}
Sep 16 18:40:56.484699 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:56.484660    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-682pg" event={"ID":"167e6fbc-ccf5-4f27-b70a-616fd4a3629e","Type":"ContainerStarted","Data":"c5580c71a402e9c3c1fbbfb0c3edb1ca13ac4460e10eb135c33384b8a85f0017"}
Sep 16 18:40:56.484844 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:56.484731    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:40:56.507012 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:56.506962    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qdskf" podStartSLOduration=65.819486351 podStartE2EDuration="1m10.506945533s" totalImagesPullingTime="4.687459182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:56.50640607 +0000 UTC m=+72.030864905" watchObservedRunningTime="2026-09-16 18:40:56.506945533 +0000 UTC m=+72.031404366"
Sep 16 18:40:56.534664 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:56.534607    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-682pg" podStartSLOduration=67.045654771 podStartE2EDuration="1m11.534586667s" totalImagesPullingTime="4.488931896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:56.531701465 +0000 UTC m=+72.056160299" watchObservedRunningTime="2026-09-16 18:40:56.534586667 +0000 UTC m=+72.059045498"
Sep 16 18:40:56.858682 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:56.858644    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:57.056944 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:57.056910    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:57.246740 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:57.246717    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:57.493507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:57.493410    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7hhn9" event={"ID":"5a09e3f9-b797-41a8-9929-4f5c73ebab63","Type":"ContainerStarted","Data":"7cf0808333bbb5cf50a3746c288b5d707226de82b7f71bac537006154099db92"}
Sep 16 18:40:57.493657 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:57.493506    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7hhn9" event={"ID":"5a09e3f9-b797-41a8-9929-4f5c73ebab63","Type":"ContainerStarted","Data":"098f14793086176ae27ea25be04c56d38b7a1e784f910fc27b0daa92fcf64c6f"}
Sep 16 18:40:57.493657 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:57.493548    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:40:57.518785 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:57.518734    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7hhn9" podStartSLOduration=33.561783528 podStartE2EDuration="37.518718061s" totalImagesPullingTime="3.956934533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:57.517908331 +0000 UTC m=+73.042367166" watchObservedRunningTime="2026-09-16 18:40:57.518718061 +0000 UTC m=+73.043176894"
Sep 16 18:40:59.976615 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:59.976569    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:40:59.979247 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:40:59.979221    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/32eb6be9-5836-434e-968e-623a8c1b4050-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-6t2vh\" (UID: \"32eb6be9-5836-434e-968e-623a8c1b4050\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:41:00.077719 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.077673    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:41:00.077900 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.077728    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:41:00.078609 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.078590    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf98bdea-b950-4609-8382-89fc440c8bdc-service-ca-bundle\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:41:00.080194 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.080178    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf98bdea-b950-4609-8382-89fc440c8bdc-metrics-certs\") pod \"router-default-69d9584dd8-d72wj\" (UID: \"cf98bdea-b950-4609-8382-89fc440c8bdc\") " pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:41:00.158584 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.158551    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mz766\""
Sep 16 18:41:00.166671 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.166651    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"
Sep 16 18:41:00.286399 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.286375    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wtd28\""
Sep 16 18:41:00.294347 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.294331    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:41:00.438585 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.438562    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh"]
Sep 16 18:41:00.440862 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:41:00.440839    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32eb6be9_5836_434e_968e_623a8c1b4050.slice/crio-3e0c5c8395e66284797ec54961e39ac0d41eecc5ae817ce0e258a979f814d1cd WatchSource:0}: Error finding container 3e0c5c8395e66284797ec54961e39ac0d41eecc5ae817ce0e258a979f814d1cd: Status 404 returned error can't find the container with id 3e0c5c8395e66284797ec54961e39ac0d41eecc5ae817ce0e258a979f814d1cd
Sep 16 18:41:00.503629 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.503601    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh" event={"ID":"32eb6be9-5836-434e-968e-623a8c1b4050","Type":"ContainerStarted","Data":"3e0c5c8395e66284797ec54961e39ac0d41eecc5ae817ce0e258a979f814d1cd"}
Sep 16 18:41:00.574034 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:00.574008    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69d9584dd8-d72wj"]
Sep 16 18:41:00.575873 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:41:00.575852    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf98bdea_b950_4609_8382_89fc440c8bdc.slice/crio-47bb1fc63590e0de7eb15017d10bd094c43b52740752be377604c0f059c51932 WatchSource:0}: Error finding container 47bb1fc63590e0de7eb15017d10bd094c43b52740752be377604c0f059c51932: Status 404 returned error can't find the container with id 47bb1fc63590e0de7eb15017d10bd094c43b52740752be377604c0f059c51932
Sep 16 18:41:01.509740 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:01.509702    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69d9584dd8-d72wj" event={"ID":"cf98bdea-b950-4609-8382-89fc440c8bdc","Type":"ContainerStarted","Data":"f497361a5e857f2cc7f493a5a8344b1b72e1bc73db5305b2cd38c4a0cba2f2b0"}
Sep 16 18:41:01.510222 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:01.509770    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69d9584dd8-d72wj" event={"ID":"cf98bdea-b950-4609-8382-89fc440c8bdc","Type":"ContainerStarted","Data":"47bb1fc63590e0de7eb15017d10bd094c43b52740752be377604c0f059c51932"}
Sep 16 18:41:01.532942 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:01.532865    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-69d9584dd8-d72wj" podStartSLOduration=33.532845589 podStartE2EDuration="33.532845589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:01.531633632 +0000 UTC m=+77.056092467" watchObservedRunningTime="2026-09-16 18:41:01.532845589 +0000 UTC m=+77.057304424"
Sep 16 18:41:02.294776 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.294728    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:41:02.297567 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.297541    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:41:02.515004 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.514911    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh" event={"ID":"32eb6be9-5836-434e-968e-623a8c1b4050","Type":"ContainerStarted","Data":"3819ca1d0285b166ba2ac51eaf3d4687918abd15d330a7744074d8b8e00c73f6"}
Sep 16 18:41:02.515461 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.515207    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:41:02.516450 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.516429    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69d9584dd8-d72wj"
Sep 16 18:41:02.538441 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.538394    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:02.539649 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.539608    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-6t2vh" podStartSLOduration=33.788498609 podStartE2EDuration="35.539592069s" totalImagesPullingTime="1.75109346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:02.538775864 +0000 UTC m=+78.063234700" watchObservedRunningTime="2026-09-16 18:41:02.539592069 +0000 UTC m=+78.064050903"
Sep 16 18:41:02.608592 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.608554    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:02.723496 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.723458    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:02.823627 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:02.823598    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:04.514575 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:04.514521    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:41:04.517077 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:04.517051    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73588ab8-b492-432b-953e-68467d462260-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tq2g6\" (UID: \"73588ab8-b492-432b-953e-68467d462260\") " pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:41:04.744750 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:04.744719    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-txfhn\""
Sep 16 18:41:04.752458 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:04.752431    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tq2g6"
Sep 16 18:41:05.035734 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:05.035701    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tq2g6"]
Sep 16 18:41:05.038784 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:41:05.038757    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73588ab8_b492_432b_953e_68467d462260.slice/crio-486eb1decd178e61ebee66337909ed670fb4bf2036fe5dc3b9a380371358f694 WatchSource:0}: Error finding container 486eb1decd178e61ebee66337909ed670fb4bf2036fe5dc3b9a380371358f694: Status 404 returned error can't find the container with id 486eb1decd178e61ebee66337909ed670fb4bf2036fe5dc3b9a380371358f694
Sep 16 18:41:05.529785 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:05.529746    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tq2g6" event={"ID":"73588ab8-b492-432b-953e-68467d462260","Type":"ContainerStarted","Data":"7b0c64cf6579679562005aea61148e9165477f06c45998cdafa80ca3c3276da5"}
Sep 16 18:41:05.530200 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:05.529819    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tq2g6" event={"ID":"73588ab8-b492-432b-953e-68467d462260","Type":"ContainerStarted","Data":"486eb1decd178e61ebee66337909ed670fb4bf2036fe5dc3b9a380371358f694"}
Sep 16 18:41:05.957540 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:05.957461    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:06.126974 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:06.126919    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:06.249700 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:06.249611    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:06.541616 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:06.541582    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tq2g6" event={"ID":"73588ab8-b492-432b-953e-68467d462260","Type":"ContainerStarted","Data":"cd06a27ba9af659d9fa5dd3a14f6d90b2830d4d173067498ecf52de7580ee278"}
Sep 16 18:41:07.499343 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:07.499313    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7hhn9"
Sep 16 18:41:07.547344 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:07.547314    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tq2g6" event={"ID":"73588ab8-b492-432b-953e-68467d462260","Type":"ContainerStarted","Data":"d5c1e12de7162c48cb955b2babe687cef96bbbc319850c37ad44023a0a7bba1b"}
Sep 16 18:41:07.589058 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:07.589000    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tq2g6" podStartSLOduration=33.773012659 podStartE2EDuration="35.588979206s" totalImagesPullingTime="1.815966547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:07.588590991 +0000 UTC m=+83.113049825" watchObservedRunningTime="2026-09-16 18:41:07.588979206 +0000 UTC m=+83.113438040"
Sep 16 18:41:09.020331 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:09.020302    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.122999 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:09.122964    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.281417 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:09.281388    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.808524 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:09.808496    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.950312 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:09.950284    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:10.103635 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:10.103540    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:12.163586 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:12.163554    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.077297 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.077264    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.199982 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.199955    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.300319 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.300283    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.579255 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.579224    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.724830 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.724797    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.751549 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.751516    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hhbjv"]
Sep 16 18:41:14.754277 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.754261    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.763758 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.763736    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:41:14.764361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.764347    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:41:14.768272 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.768252    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:41:14.770843 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.770826    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:41:14.775610 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.775583    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cxq8v\""
Sep 16 18:41:14.894017 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.893923    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-metrics-client-ca\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.894192 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.894021    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-textfile\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.894192 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.894064    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-accelerators-collector-config\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.894192 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.894119    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.894348 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.894213    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-wtmp\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.894348 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.894249    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2kw\" (UniqueName: \"kubernetes.io/projected/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-kube-api-access-mb2kw\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.894348 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.894296    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-root\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.894464 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.894353    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-tls\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.894464 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.894392    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-sys\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.899279 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.899257    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.995658 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995622    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-metrics-client-ca\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.995658 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995661    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-textfile\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.995926 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995683    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-accelerators-collector-config\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.995926 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995703    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.995926 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995802    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-wtmp\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.995926 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995845    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2kw\" (UniqueName: \"kubernetes.io/projected/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-kube-api-access-mb2kw\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.995926 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995909    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-root\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.996184 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995934    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-tls\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.996184 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995953    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-sys\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.996184 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.995953    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-wtmp\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.996184 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.996038    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-root\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.996184 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.996077    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-sys\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.996184 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.996086    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-textfile\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.996507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.996320    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-metrics-client-ca\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.996507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.996393    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-accelerators-collector-config\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.998284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.998266    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:14.998377 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:14.998358    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-node-exporter-tls\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:15.029242 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:15.029211    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2kw\" (UniqueName: \"kubernetes.io/projected/bb5a3fd8-458f-43ef-b9b9-75f1303197c1-kube-api-access-mb2kw\") pod \"node-exporter-hhbjv\" (UID: \"bb5a3fd8-458f-43ef-b9b9-75f1303197c1\") " pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:15.035282 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:15.035257    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:15.062897 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:15.062864    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hhbjv"
Sep 16 18:41:15.074820 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:41:15.074795    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5a3fd8_458f_43ef_b9b9_75f1303197c1.slice/crio-20a2f466731410bfa451debc4734a7dcf0c730d7b85f462ddbbd8820770084f5 WatchSource:0}: Error finding container 20a2f466731410bfa451debc4734a7dcf0c730d7b85f462ddbbd8820770084f5: Status 404 returned error can't find the container with id 20a2f466731410bfa451debc4734a7dcf0c730d7b85f462ddbbd8820770084f5
Sep 16 18:41:15.169755 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:15.169669    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:15.449977 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:15.449852    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:15.574087 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:15.574053    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hhbjv" event={"ID":"bb5a3fd8-458f-43ef-b9b9-75f1303197c1","Type":"ContainerStarted","Data":"20a2f466731410bfa451debc4734a7dcf0c730d7b85f462ddbbd8820770084f5"}
Sep 16 18:41:15.751013 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:15.750931    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:15.889754 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:15.889711    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:16.144551 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:16.144508    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:16.315103 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:16.315066    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:16.917633 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:16.917536    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.033053 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:17.033024    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.582084 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:17.582048    2590 generic.go:356] "Generic (PLEG): container finished" podID="bb5a3fd8-458f-43ef-b9b9-75f1303197c1" containerID="495357e8bbf44365ce1ac68ae33bb6e6e05a7fcef9a043aa371dd1aaaf50025d" exitCode=0
Sep 16 18:41:17.582455 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:17.582116    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hhbjv" event={"ID":"bb5a3fd8-458f-43ef-b9b9-75f1303197c1","Type":"ContainerDied","Data":"495357e8bbf44365ce1ac68ae33bb6e6e05a7fcef9a043aa371dd1aaaf50025d"}
Sep 16 18:41:18.108528 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:18.108482    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:18.235471 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:18.235436    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:18.486672 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:18.486585    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:18.574754 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:18.574718    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:18.587573 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:18.587543    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hhbjv" event={"ID":"bb5a3fd8-458f-43ef-b9b9-75f1303197c1","Type":"ContainerStarted","Data":"328dfbb87711285d0650e4ed7cff5e7551d3bfb81d622fbbd0c57b36a60ce37f"}
Sep 16 18:41:18.587925 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:18.587590    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hhbjv" event={"ID":"bb5a3fd8-458f-43ef-b9b9-75f1303197c1","Type":"ContainerStarted","Data":"5868d43ebb0ef5e3fc6cb9551b191e5d20229a214f90d229328a4291fb7ef168"}
Sep 16 18:41:18.616359 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:18.616310    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hhbjv" podStartSLOduration=2.976867236 podStartE2EDuration="4.616292365s" totalImagesPullingTime="1.542302613s" totalInitContainerRuntime="97.122516ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:18.615424872 +0000 UTC m=+94.139883717" watchObservedRunningTime="2026-09-16 18:41:18.616292365 +0000 UTC m=+94.140751237"
Sep 16 18:41:19.161231 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:19.161202    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.399149 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:19.399114    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:21.602917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:21.602859    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.069772 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:23.069699    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.179716 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:23.179679    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.280465 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:23.280428    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.435295 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:23.435210    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.666140 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:23.666101    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:23.789073 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:23.789037    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:24.056380 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:24.056304    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:24.142699 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:24.142665    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:24.324390 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:24.324313    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:24.428763 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:24.428730    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:24.834670 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:24.834640    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:25.043623 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:25.043592    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:25.602186 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:25.602154    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:25.894285 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:25.894210    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:26.003381 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:26.003348    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:26.090134 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:26.090096    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:26.305238 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:26.305209    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:26.395798 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:26.395763    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:27.495944 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:27.495917    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-682pg"
Sep 16 18:41:28.949716 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:28.949680    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:29.042988 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:29.042952    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:30.917455 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:30.917419    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:31.071314 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:31.071277    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:31.239807 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:31.239737    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:31.370350 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:31.370317    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:31.505360 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:31.505291    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:32.803606 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:32.803550    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:32.956047 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:32.956017    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:38.266514 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:38.266483    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:38.825415 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:38.825387    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:38.976469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:38.976429    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.224835 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:40.224800    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.513363 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:40.513274    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.634414 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:40.634380    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.113454 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:41.113415    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.208594 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:41.208555    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.901980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:41.901939    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:42.008579 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:42.008548    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.660201 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:43.660165    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.761366 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:43.761336    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.107935 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:44.107868    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.177502 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:44.177469    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.693816 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:47.693781    2590 generic.go:356] "Generic (PLEG): container finished" podID="c2e9887b-61ad-49be-9252-a113339cd4ff" containerID="605536a8f788765c99d68e4f7a2983642b7c27ca11d44792791ed18872489d43" exitCode=0
Sep 16 18:41:47.694192 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:47.693847    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv" event={"ID":"c2e9887b-61ad-49be-9252-a113339cd4ff","Type":"ContainerDied","Data":"605536a8f788765c99d68e4f7a2983642b7c27ca11d44792791ed18872489d43"}
Sep 16 18:41:47.694240 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:47.694196    2590 scope.go:117] "RemoveContainer" containerID="605536a8f788765c99d68e4f7a2983642b7c27ca11d44792791ed18872489d43"
Sep 16 18:41:48.699076 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:48.699043    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-b68lv" event={"ID":"c2e9887b-61ad-49be-9252-a113339cd4ff","Type":"ContainerStarted","Data":"5a9a87ac0adb68e266db7748a4e2753ac189cfc8997eebef5839a8c7a8402f7d"}
Sep 16 18:41:50.450110 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:50.450074    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:50.574372 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:50.574340    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:50.902756 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:50.902724    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:50.984390 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:50.984353    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:52.161091 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:52.161056    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:52.270057 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:52.270024    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:52.389288 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:52.389249    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:52.676069 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:52.676034    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:52.778870 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:52.778841    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:54.806532 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:54.806492    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:41:54.808954 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:54.808929    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5989f247-505b-4069-92dc-8d563bfeaae6-metrics-certs\") pod \"network-metrics-daemon-pg5zg\" (UID: \"5989f247-505b-4069-92dc-8d563bfeaae6\") " pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:41:54.934115 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:54.934082    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcswh\""
Sep 16 18:41:54.942381 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:54.942361    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pg5zg"
Sep 16 18:41:55.226389 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:55.226306    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pg5zg"]
Sep 16 18:41:55.229167 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:41:55.229142    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5989f247_505b_4069_92dc_8d563bfeaae6.slice/crio-23dee64728b0ab5b1df7a5270d72bf2e5e4d1186bde663e8621cf5b31cdabb16 WatchSource:0}: Error finding container 23dee64728b0ab5b1df7a5270d72bf2e5e4d1186bde663e8621cf5b31cdabb16: Status 404 returned error can't find the container with id 23dee64728b0ab5b1df7a5270d72bf2e5e4d1186bde663e8621cf5b31cdabb16
Sep 16 18:41:55.722996 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:55.722929    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pg5zg" event={"ID":"5989f247-505b-4069-92dc-8d563bfeaae6","Type":"ContainerStarted","Data":"23dee64728b0ab5b1df7a5270d72bf2e5e4d1186bde663e8621cf5b31cdabb16"}
Sep 16 18:41:56.728022 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:56.727984    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pg5zg" event={"ID":"5989f247-505b-4069-92dc-8d563bfeaae6","Type":"ContainerStarted","Data":"70e3cb1796ecfb6a31b30d2315f92456cca776cc8b0728a6c2433efb5117537e"}
Sep 16 18:41:56.728415 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:56.728079    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pg5zg" event={"ID":"5989f247-505b-4069-92dc-8d563bfeaae6","Type":"ContainerStarted","Data":"8805b953bbc38acc7683b9a0bbfd05ed5123d107860271fe214bb84e6675ec23"}
Sep 16 18:41:56.756706 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:56.756653    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pg5zg" podStartSLOduration=130.961702856 podStartE2EDuration="2m11.756637261s" totalImagesPullingTime="794.934405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:56.755356058 +0000 UTC m=+132.279814893" watchObservedRunningTime="2026-09-16 18:41:56.756637261 +0000 UTC m=+132.281096094"
Sep 16 18:41:56.897452 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:56.897417    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:56.968988 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:56.968948    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:59.102474 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:59.102435    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:59.210919 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:41:59.210869    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:01.043847 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:01.043789    2590 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" podUID="8700d590-924f-4777-9234-564844aba86b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:42:01.747536 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:01.747503    2590 generic.go:356] "Generic (PLEG): container finished" podID="5da5b179-92bd-4308-a524-a08566a60d2d" containerID="43a1c70c921514e43a9db423a4d34a3a5b82f0b94f91ffe3d171c5288168286b" exitCode=0
Sep 16 18:42:01.747536 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:01.747539    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2" event={"ID":"5da5b179-92bd-4308-a524-a08566a60d2d","Type":"ContainerDied","Data":"43a1c70c921514e43a9db423a4d34a3a5b82f0b94f91ffe3d171c5288168286b"}
Sep 16 18:42:01.747832 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:01.747821    2590 scope.go:117] "RemoveContainer" containerID="43a1c70c921514e43a9db423a4d34a3a5b82f0b94f91ffe3d171c5288168286b"
Sep 16 18:42:02.753104 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:02.753059    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5lrj2" event={"ID":"5da5b179-92bd-4308-a524-a08566a60d2d","Type":"ContainerStarted","Data":"d88022b2d893a919a6abbc4499966aaa248405cb8031b0bef9af6cdfeb387c2d"}
Sep 16 18:42:02.754546 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:02.754523    2590 generic.go:356] "Generic (PLEG): container finished" podID="eb13daed-c369-49b7-9e38-bdd0636bd0ad" containerID="7f87f3d254184d9337625d82e77117d65da1fae31ee6b770e8f39cdfba030143" exitCode=0
Sep 16 18:42:02.754656 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:02.754594    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h6hs2" event={"ID":"eb13daed-c369-49b7-9e38-bdd0636bd0ad","Type":"ContainerDied","Data":"7f87f3d254184d9337625d82e77117d65da1fae31ee6b770e8f39cdfba030143"}
Sep 16 18:42:02.754911 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:02.754874    2590 scope.go:117] "RemoveContainer" containerID="7f87f3d254184d9337625d82e77117d65da1fae31ee6b770e8f39cdfba030143"
Sep 16 18:42:03.760786 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:03.760749    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h6hs2" event={"ID":"eb13daed-c369-49b7-9e38-bdd0636bd0ad","Type":"ContainerStarted","Data":"ee2bba3467b267762f0ffa9a8e1038243b6cd13261b93c1403b67b8ae8674f53"}
Sep 16 18:42:07.249219 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:07.249183    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:11.043981 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:11.043940    2590 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" podUID="8700d590-924f-4777-9234-564844aba86b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:42:14.103711 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:14.103679    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:14.174430 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:14.174389    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:21.043591 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:21.043548    2590 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" podUID="8700d590-924f-4777-9234-564844aba86b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:42:21.044080 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:21.043628    2590 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8"
Sep 16 18:42:21.044246 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:21.044209    2590 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c79b7692ba8e7d61017d704aa06c5dd6785b183911f204f417639b29eb90ada4"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 18:42:21.044322 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:21.044277    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" podUID="8700d590-924f-4777-9234-564844aba86b" containerName="service-proxy" containerID="cri-o://c79b7692ba8e7d61017d704aa06c5dd6785b183911f204f417639b29eb90ada4" gracePeriod=30
Sep 16 18:42:21.826372 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:21.826334    2590 generic.go:356] "Generic (PLEG): container finished" podID="8700d590-924f-4777-9234-564844aba86b" containerID="c79b7692ba8e7d61017d704aa06c5dd6785b183911f204f417639b29eb90ada4" exitCode=2
Sep 16 18:42:21.826539 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:21.826405    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" event={"ID":"8700d590-924f-4777-9234-564844aba86b","Type":"ContainerDied","Data":"c79b7692ba8e7d61017d704aa06c5dd6785b183911f204f417639b29eb90ada4"}
Sep 16 18:42:21.826539 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:21.826452    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55fb4fc48c-lkvm8" event={"ID":"8700d590-924f-4777-9234-564844aba86b","Type":"ContainerStarted","Data":"68907357e9dfd71ba7ad982845fa580fcb07b6159a85d64ac80e01b459f3b6ae"}
Sep 16 18:42:24.989755 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:24.989722    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:31.377202 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:31.377167    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:31.470615 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:31.470580    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:43.337580 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:43.337545    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:43.439936 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:43.439898    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:56.807139 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:56.807107    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:56.882082 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:56.882043    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:59.125031 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:59.124990    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:59.224362 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:42:59.224331    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:07.384328 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:43:07.384295    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:07.485404 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:43:07.485367    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:07.587350 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:43:07.587309    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:33.996297 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:43:33.996258    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:34.069206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:43:34.069169    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:50.364859 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:43:50.364824    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:50.464219 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:43:50.464188    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:50.567271 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:43:50.567224    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:16.732447 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:16.732350    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:16.827859 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:16.827829    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:18.394443 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:18.394403    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:18.502145 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:18.502101    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:22.210454 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:22.210416    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:22.282986 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:22.282955    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:35.596587 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.596542    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:35.885723 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.885631    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:35.905407 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.905372    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5qnlv"]
Sep 16 18:44:35.908230 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.908208    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:35.911273 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.911251    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:44:35.912093 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.912064    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:44:35.912207 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.912118    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:44:35.912207 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.912124    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:44:35.912476 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.912460    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7sl84\""
Sep 16 18:44:35.912619 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.912606    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:44:35.920343 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.920317    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5qnlv"]
Sep 16 18:44:35.987155 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:35.987122    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.061143 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.061110    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-certificates\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:36.061143 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.061146    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3af95d7d-3e39-44d0-be71-8b85f20adbb8-cabundle0\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:36.061289 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.061172    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbl8\" (UniqueName: \"kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-kube-api-access-cnbl8\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:36.161637 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.161546    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-certificates\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:36.161637 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.161594    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3af95d7d-3e39-44d0-be71-8b85f20adbb8-cabundle0\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:36.161637 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.161616    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbl8\" (UniqueName: \"kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-kube-api-access-cnbl8\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:36.161832 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:44:36.161711    2590 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:44:36.161832 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:44:36.161732    2590 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:44:36.161832 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:44:36.161741    2590 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5qnlv: references non-existent secret key: ca.crt
Sep 16 18:44:36.161832 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:44:36.161807    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-certificates podName:3af95d7d-3e39-44d0-be71-8b85f20adbb8 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:36.661789615 +0000 UTC m=+292.186248430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-certificates") pod "keda-operator-579c8958f7-5qnlv" (UID: "3af95d7d-3e39-44d0-be71-8b85f20adbb8") : references non-existent secret key: ca.crt
Sep 16 18:44:36.162356 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.162338    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3af95d7d-3e39-44d0-be71-8b85f20adbb8-cabundle0\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:36.180645 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.180610    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbl8\" (UniqueName: \"kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-kube-api-access-cnbl8\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:36.539609 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.539580    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.648351 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.648311    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.665474 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:36.665445    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-certificates\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:36.665612 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:44:36.665562    2590 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:44:36.665612 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:44:36.665575    2590 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:44:36.665612 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:44:36.665583    2590 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5qnlv: references non-existent secret key: ca.crt
Sep 16 18:44:36.665734 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:44:36.665637    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-certificates podName:3af95d7d-3e39-44d0-be71-8b85f20adbb8 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:37.665624055 +0000 UTC m=+293.190082867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-certificates") pod "keda-operator-579c8958f7-5qnlv" (UID: "3af95d7d-3e39-44d0-be71-8b85f20adbb8") : references non-existent secret key: ca.crt
Sep 16 18:44:37.361663 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:37.361617    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.519755 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:37.519700    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.622733 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:37.622638    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.673777 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:37.673735    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-certificates\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:37.676324 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:37.676298    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3af95d7d-3e39-44d0-be71-8b85f20adbb8-certificates\") pod \"keda-operator-579c8958f7-5qnlv\" (UID: \"3af95d7d-3e39-44d0-be71-8b85f20adbb8\") " pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:37.717856 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:37.717816    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:37.883579 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:37.883499    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.994435 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:37.994412    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5qnlv"]
Sep 16 18:44:37.998398 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:44:37.998354    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af95d7d_3e39_44d0_be71_8b85f20adbb8.slice/crio-ab6cf5703c114e342415e2c89e4c7db6fc2cf3cdc80160fdc77e350f4653f14b WatchSource:0}: Error finding container ab6cf5703c114e342415e2c89e4c7db6fc2cf3cdc80160fdc77e350f4653f14b: Status 404 returned error can't find the container with id ab6cf5703c114e342415e2c89e4c7db6fc2cf3cdc80160fdc77e350f4653f14b
Sep 16 18:44:38.089633 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:38.089602    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:38.287260 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:38.287224    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5qnlv" event={"ID":"3af95d7d-3e39-44d0-be71-8b85f20adbb8","Type":"ContainerStarted","Data":"ab6cf5703c114e342415e2c89e4c7db6fc2cf3cdc80160fdc77e350f4653f14b"}
Sep 16 18:44:38.292661 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:38.292634    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:38.390470 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:38.390434    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:38.678596 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:38.678514    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:38.786070 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:38.786038    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.306830 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:42.306791    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5qnlv" event={"ID":"3af95d7d-3e39-44d0-be71-8b85f20adbb8","Type":"ContainerStarted","Data":"d75b02e1145aa8bca5dee9ff4203541cca2d976fc42ff5404826d26b69abebf7"}
Sep 16 18:44:42.307284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:42.306905    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:44:42.323780 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:42.323715    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-5qnlv" podStartSLOduration=3.7861298039999998 podStartE2EDuration="7.32369479s" totalImagesPullingTime="3.537564986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:42.323401847 +0000 UTC m=+297.847860683" watchObservedRunningTime="2026-09-16 18:44:42.32369479 +0000 UTC m=+297.848153627"
Sep 16 18:44:44.884505 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:44:44.884471    2590 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:03.313363 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:03.313332    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-5qnlv"
Sep 16 18:45:27.256751 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:27.256717    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:29.256812 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:29.256783    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:38.691516 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:38.691423    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:38.980064 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:38.979979    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:41.259930 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.258313    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"]
Sep 16 18:45:41.261256 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.261233    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"
Sep 16 18:45:41.264123 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.264100    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zgj6g\""
Sep 16 18:45:41.264236 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.264106    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:41.264294 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.264245    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:41.271692 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.271672    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"]
Sep 16 18:45:41.332625 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.332586    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ef9c81-9118-4860-971e-3590d7bf727e-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-flf94\" (UID: \"d5ef9c81-9118-4860-971e-3590d7bf727e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"
Sep 16 18:45:41.332625 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.332622    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwdd\" (UniqueName: \"kubernetes.io/projected/d5ef9c81-9118-4860-971e-3590d7bf727e-kube-api-access-mbwdd\") pod \"cert-manager-operator-controller-manager-684f79c9b9-flf94\" (UID: \"d5ef9c81-9118-4860-971e-3590d7bf727e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"
Sep 16 18:45:41.433608 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.433573    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ef9c81-9118-4860-971e-3590d7bf727e-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-flf94\" (UID: \"d5ef9c81-9118-4860-971e-3590d7bf727e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"
Sep 16 18:45:41.433608 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.433609    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwdd\" (UniqueName: \"kubernetes.io/projected/d5ef9c81-9118-4860-971e-3590d7bf727e-kube-api-access-mbwdd\") pod \"cert-manager-operator-controller-manager-684f79c9b9-flf94\" (UID: \"d5ef9c81-9118-4860-971e-3590d7bf727e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"
Sep 16 18:45:41.434014 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.433994    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ef9c81-9118-4860-971e-3590d7bf727e-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-flf94\" (UID: \"d5ef9c81-9118-4860-971e-3590d7bf727e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"
Sep 16 18:45:41.443986 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.443961    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwdd\" (UniqueName: \"kubernetes.io/projected/d5ef9c81-9118-4860-971e-3590d7bf727e-kube-api-access-mbwdd\") pod \"cert-manager-operator-controller-manager-684f79c9b9-flf94\" (UID: \"d5ef9c81-9118-4860-971e-3590d7bf727e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"
Sep 16 18:45:41.570584 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.570547    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"
Sep 16 18:45:41.862356 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.862275    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94"]
Sep 16 18:45:41.865729 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:45:41.865706    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ef9c81_9118_4860_971e_3590d7bf727e.slice/crio-f8173c0a1ff44582c8ba210885070119e8ba2b99662b21961f1755df1d5546a1 WatchSource:0}: Error finding container f8173c0a1ff44582c8ba210885070119e8ba2b99662b21961f1755df1d5546a1: Status 404 returned error can't find the container with id f8173c0a1ff44582c8ba210885070119e8ba2b99662b21961f1755df1d5546a1
Sep 16 18:45:41.869208 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:41.869189    2590 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:42.511842 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:42.511806    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94" event={"ID":"d5ef9c81-9118-4860-971e-3590d7bf727e","Type":"ContainerStarted","Data":"f8173c0a1ff44582c8ba210885070119e8ba2b99662b21961f1755df1d5546a1"}
Sep 16 18:45:42.876040 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:42.875992    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:42.979952 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:42.979920    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.529329 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:45.529293    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94" event={"ID":"d5ef9c81-9118-4860-971e-3590d7bf727e","Type":"ContainerStarted","Data":"29cb3eb87825247dde990744ed5aec92c8aa665188339649f9373a839fc425ae"}
Sep 16 18:45:45.560420 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:45.560337    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-flf94" podStartSLOduration=1.6740226919999999 podStartE2EDuration="4.560314494s" totalImagesPullingTime="2.886291802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:45.558135772 +0000 UTC m=+361.082594608" watchObservedRunningTime="2026-09-16 18:45:45.560314494 +0000 UTC m=+361.084773329"
Sep 16 18:45:45.615371 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:45.615338    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.712269 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:45.712236    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:47.833173 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:47.833143    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.002829 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:48.002794    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.102577 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:48.102499    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.202723 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:48.202693    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.687985 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:48.687957    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.794997 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:48.794964    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.890339 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:48.890304    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:55.435770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:55.435723    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:55.535654 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:55.535621    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:56.416685 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:45:56.416654    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:00.960071 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:00.960035    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:01.060645 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:01.060616    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:02.326683 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.326650    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"]
Sep 16 18:46:02.329919 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.329898    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"
Sep 16 18:46:02.332443 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.332423    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-j6dzx\""
Sep 16 18:46:02.332443 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.332430    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:02.333465 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.333442    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:02.340171 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.340145    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"]
Sep 16 18:46:02.405224 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.405176    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72131bdb-e31a-4477-bbaa-694c991657fd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-h5vpr\" (UID: \"72131bdb-e31a-4477-bbaa-694c991657fd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"
Sep 16 18:46:02.405413 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.405255    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df8kj\" (UniqueName: \"kubernetes.io/projected/72131bdb-e31a-4477-bbaa-694c991657fd-kube-api-access-df8kj\") pod \"openshift-lws-operator-fd8ccff4c-h5vpr\" (UID: \"72131bdb-e31a-4477-bbaa-694c991657fd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"
Sep 16 18:46:02.506539 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.506501    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df8kj\" (UniqueName: \"kubernetes.io/projected/72131bdb-e31a-4477-bbaa-694c991657fd-kube-api-access-df8kj\") pod \"openshift-lws-operator-fd8ccff4c-h5vpr\" (UID: \"72131bdb-e31a-4477-bbaa-694c991657fd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"
Sep 16 18:46:02.506539 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.506545    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72131bdb-e31a-4477-bbaa-694c991657fd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-h5vpr\" (UID: \"72131bdb-e31a-4477-bbaa-694c991657fd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"
Sep 16 18:46:02.506962 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.506946    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72131bdb-e31a-4477-bbaa-694c991657fd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-h5vpr\" (UID: \"72131bdb-e31a-4477-bbaa-694c991657fd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"
Sep 16 18:46:02.515312 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.515288    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df8kj\" (UniqueName: \"kubernetes.io/projected/72131bdb-e31a-4477-bbaa-694c991657fd-kube-api-access-df8kj\") pod \"openshift-lws-operator-fd8ccff4c-h5vpr\" (UID: \"72131bdb-e31a-4477-bbaa-694c991657fd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"
Sep 16 18:46:02.638962 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.638847    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"
Sep 16 18:46:02.911579 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:02.911468    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr"]
Sep 16 18:46:02.914297 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:46:02.914267    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72131bdb_e31a_4477_bbaa_694c991657fd.slice/crio-b9d4eb7a6e5da51d7b3ef7d6f1800c6cd0b412009393bf878153a63d46719af7 WatchSource:0}: Error finding container b9d4eb7a6e5da51d7b3ef7d6f1800c6cd0b412009393bf878153a63d46719af7: Status 404 returned error can't find the container with id b9d4eb7a6e5da51d7b3ef7d6f1800c6cd0b412009393bf878153a63d46719af7
Sep 16 18:46:03.604234 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:03.604194    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr" event={"ID":"72131bdb-e31a-4477-bbaa-694c991657fd","Type":"ContainerStarted","Data":"b9d4eb7a6e5da51d7b3ef7d6f1800c6cd0b412009393bf878153a63d46719af7"}
Sep 16 18:46:05.616507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:05.616462    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr" event={"ID":"72131bdb-e31a-4477-bbaa-694c991657fd","Type":"ContainerStarted","Data":"8a4e5d99bf1a9668fbc38c224beb2e6f469afcca171c1771ffdb3bae308c434e"}
Sep 16 18:46:05.634863 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:05.634802    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h5vpr" podStartSLOduration=1.032116258 podStartE2EDuration="3.634781999s" totalImagesPullingTime="2.602665741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:05.633612421 +0000 UTC m=+381.158071256" watchObservedRunningTime="2026-09-16 18:46:05.634781999 +0000 UTC m=+381.159240815"
Sep 16 18:46:12.148244 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:12.148205    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:17.363290 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:17.363249    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:17.465782 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:17.465738    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:18.606015 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:18.605978    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.105752 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:21.105718    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.197495 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:21.197463    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.696075 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.696048    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"]
Sep 16 18:46:23.701702 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.701679    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.704034 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.704011    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:46:23.704260 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.704241    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-q2wx5\""
Sep 16 18:46:23.721123 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.721100    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"]
Sep 16 18:46:23.741700 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.741676    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.772391 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.772363    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.772534 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.772397    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.772534 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.772424    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.772534 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.772469    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.772534 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.772490    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5j8s\" (UniqueName: \"kubernetes.io/projected/a9490cec-b84b-4cc7-aefa-946d4e5646fb-kube-api-access-q5j8s\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.772534 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.772528    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.772711 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.772548    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.772711 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.772567    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.772711 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.772583    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.826282 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.826253    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.872967 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.872935    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5j8s\" (UniqueName: \"kubernetes.io/projected/a9490cec-b84b-4cc7-aefa-946d4e5646fb-kube-api-access-q5j8s\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873105 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.872989    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873105 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873027    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873105 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873058    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873105 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873085    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873290 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873113    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873290 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873138    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873290 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873184    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873290 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873225    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873506 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873483    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873617 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873600    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873727 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873706    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.873775 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.873716    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.874252 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.874229    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.875655 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.875636    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.875792 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.875775    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.881488 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.881464    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9490cec-b84b-4cc7-aefa-946d4e5646fb-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:23.881751 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:23.881727    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5j8s\" (UniqueName: \"kubernetes.io/projected/a9490cec-b84b-4cc7-aefa-946d4e5646fb-kube-api-access-q5j8s\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-blgj4\" (UID: \"a9490cec-b84b-4cc7-aefa-946d4e5646fb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:24.013280 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:24.013178    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:24.294466 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:24.294441    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"]
Sep 16 18:46:24.296833 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:46:24.296809    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9490cec_b84b_4cc7_aefa_946d4e5646fb.slice/crio-f61d01a3cb20768dd9bb7ab204ce303d02bc2f651599eeb36fd598c8313930e1 WatchSource:0}: Error finding container f61d01a3cb20768dd9bb7ab204ce303d02bc2f651599eeb36fd598c8313930e1: Status 404 returned error can't find the container with id f61d01a3cb20768dd9bb7ab204ce303d02bc2f651599eeb36fd598c8313930e1
Sep 16 18:46:24.690873 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:24.690788    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" event={"ID":"a9490cec-b84b-4cc7-aefa-946d4e5646fb","Type":"ContainerStarted","Data":"f61d01a3cb20768dd9bb7ab204ce303d02bc2f651599eeb36fd598c8313930e1"}
Sep 16 18:46:26.420794 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:26.420758    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:26.918772 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:26.918738    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:46:26.918853 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:26.918810    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:46:26.918853 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:26.918840    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:46:27.708609 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:27.708566    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" event={"ID":"a9490cec-b84b-4cc7-aefa-946d4e5646fb","Type":"ContainerStarted","Data":"ae6b01464eaae175b3dffa1c2f51464cd6d0574e77c38e126c6edd1607bf5853"}
Sep 16 18:46:27.734254 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:27.734192    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" podStartSLOduration=2.114900865 podStartE2EDuration="4.734175188s" totalImagesPullingTime="2.619274323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:27.732150107 +0000 UTC m=+403.256608941" watchObservedRunningTime="2026-09-16 18:46:27.734175188 +0000 UTC m=+403.258634021"
Sep 16 18:46:28.013805 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:28.013666    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:28.018365 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:28.018335    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:28.713045 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:28.713013    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:28.714197 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:28.714177    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:46:29.066378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:29.066347    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.264170 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:29.264135    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.381768 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:29.381663    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.694324 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.694295    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.787932 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.787896    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.803670 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.803639    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"]
Sep 16 18:46:30.806869 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.806853    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.809645 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.809625    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:46:30.810377 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.810351    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:46:30.810377 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.810353    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8fq9q\""
Sep 16 18:46:30.810540 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.810420    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:46:30.820592 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.820570    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"]
Sep 16 18:46:30.825590 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.825568    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-metrics-cert\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.825701 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.825663    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-manager-config\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.825822 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.825804    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfq9\" (UniqueName: \"kubernetes.io/projected/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-kube-api-access-4cfq9\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.825922 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.825832    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-cert\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.916507 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.916468    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.926646 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.926617    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-metrics-cert\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.926787 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.926669    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-manager-config\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.926787 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.926694    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfq9\" (UniqueName: \"kubernetes.io/projected/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-kube-api-access-4cfq9\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.926787 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.926712    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-cert\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.927339 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.927315    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-manager-config\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.929468 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.929448    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-metrics-cert\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.929548 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.929451    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-cert\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:30.953354 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:30.953290    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfq9\" (UniqueName: \"kubernetes.io/projected/d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35-kube-api-access-4cfq9\") pod \"lws-controller-manager-856c96b694-9x9nz\" (UID: \"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:31.116593 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:31.116556    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:31.118234 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:31.118194    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.220557 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:31.220472    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.388595 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:31.388570    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"]
Sep 16 18:46:31.390944 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:46:31.390914    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76c8eb5_fb50_45f4_a2b5_8dfb0804ca35.slice/crio-c09e24c2a994a95275b60e0fb3cc00fb73955c8a59af87038b64c8a11bd9083e WatchSource:0}: Error finding container c09e24c2a994a95275b60e0fb3cc00fb73955c8a59af87038b64c8a11bd9083e: Status 404 returned error can't find the container with id c09e24c2a994a95275b60e0fb3cc00fb73955c8a59af87038b64c8a11bd9083e
Sep 16 18:46:31.727437 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:31.727403    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz" event={"ID":"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35","Type":"ContainerStarted","Data":"c09e24c2a994a95275b60e0fb3cc00fb73955c8a59af87038b64c8a11bd9083e"}
Sep 16 18:46:37.757045 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:37.756949    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz" event={"ID":"d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35","Type":"ContainerStarted","Data":"fb786ad0e2cd07925430f02e648db0af15b11abb9af625d97d5eea851fc19123"}
Sep 16 18:46:37.757445 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:37.757093    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:37.776545 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:37.776496    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz" podStartSLOduration=1.876346679 podStartE2EDuration="7.776480374s" totalImagesPullingTime="5.900133695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:37.775527664 +0000 UTC m=+413.299986499" watchObservedRunningTime="2026-09-16 18:46:37.776480374 +0000 UTC m=+413.300939277"
Sep 16 18:46:40.150579 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:40.150506    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.218985 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:40.218935    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.356922 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:44.356853    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:45.167455 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:45.167423    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:45.940297 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:45.940229    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:48.763051 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:48.763017    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-856c96b694-9x9nz"
Sep 16 18:46:49.323276 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:49.323226    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:50.911466 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:50.911414    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.262178 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:51.262043    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.356259 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:51.356215    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.649104 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:51.649048    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.919614 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:51.919527    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:52.064648 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:52.064599    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:52.178563 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:52.178435    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:53.575492 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:53.575456    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:53.666917 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:53.666873    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:54.338591 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:54.338510    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:55.016308 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.016272    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:55.781414 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.781381    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:55.805277 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.805248    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"]
Sep 16 18:46:55.811523 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.811502    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:55.814894 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.814862    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 18:46:55.814986 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.814934    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 18:46:55.815437 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.815421    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:46:55.815842 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.815829    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:46:55.816005 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.815991    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-48428\""
Sep 16 18:46:55.828574 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.828552    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"]
Sep 16 18:46:55.927867 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.927833    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4rv8\" (UniqueName: \"kubernetes.io/projected/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-kube-api-access-d4rv8\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:55.928061 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.927952    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:55.928061 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:55.927997    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:56.028495 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:56.028464    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:56.028918 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:56.028510    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:56.028918 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:56.028543    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4rv8\" (UniqueName: \"kubernetes.io/projected/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-kube-api-access-d4rv8\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:56.028918 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:46:56.028618    2590 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 18:46:56.028918 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:46:56.028699    2590 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-plugin-serving-cert podName:5620cf6b-409b-4198-a9b7-0a4d0351c6b9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:56.528681235 +0000 UTC m=+432.053140067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-s4vqj" (UID: "5620cf6b-409b-4198-a9b7-0a4d0351c6b9") : secret "plugin-serving-cert" not found
Sep 16 18:46:56.029274 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:56.029256    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:56.068090 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:56.068015    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4rv8\" (UniqueName: \"kubernetes.io/projected/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-kube-api-access-d4rv8\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:56.515515 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:56.515406    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.531659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:56.531629    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:56.534238 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:56.534217    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5620cf6b-409b-4198-a9b7-0a4d0351c6b9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-s4vqj\" (UID: \"5620cf6b-409b-4198-a9b7-0a4d0351c6b9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:56.720658 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:56.720619    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"
Sep 16 18:46:57.003827 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:57.003793    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj"]
Sep 16 18:46:57.007386 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:46:57.007356    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5620cf6b_409b_4198_a9b7_0a4d0351c6b9.slice/crio-892e4d75b0853eff78278f133837c47d1b663702f6b86a86f6ae24461ffb2be8 WatchSource:0}: Error finding container 892e4d75b0853eff78278f133837c47d1b663702f6b86a86f6ae24461ffb2be8: Status 404 returned error can't find the container with id 892e4d75b0853eff78278f133837c47d1b663702f6b86a86f6ae24461ffb2be8
Sep 16 18:46:57.841473 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:57.841435    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj" event={"ID":"5620cf6b-409b-4198-a9b7-0a4d0351c6b9","Type":"ContainerStarted","Data":"892e4d75b0853eff78278f133837c47d1b663702f6b86a86f6ae24461ffb2be8"}
Sep 16 18:46:59.347405 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:59.347365    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.427920 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:59.427861    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.736151 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:59.736051    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.837952 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:46:59.837910    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:00.137281 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:00.137076    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:00.239373 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:00.239319    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:01.861854 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:01.861765    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj" event={"ID":"5620cf6b-409b-4198-a9b7-0a4d0351c6b9","Type":"ContainerStarted","Data":"4cb8c11533247c837d2f578bfe5d3179aecefbe0e2ef257feb5f62f107dfbd3b"}
Sep 16 18:47:01.882743 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:01.882680    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-s4vqj" podStartSLOduration=2.2930477160000002 podStartE2EDuration="6.882663524s" totalImagesPullingTime="4.589615808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.881654156 +0000 UTC m=+437.406112992" watchObservedRunningTime="2026-09-16 18:47:01.882663524 +0000 UTC m=+437.407122358"
Sep 16 18:47:06.383069 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:06.382973    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:06.484261 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:06.484229    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:11.152779 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:11.152745    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:11.293613 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:11.293576    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:13.634758 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:13.634725    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:13.763665 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:13.763634    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:19.857223 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:19.857191    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:20.572193 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:20.572147    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:20.675291 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:20.675242    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:20.992909 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:20.992803    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:21.172550 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:21.172517    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:21.278207 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:21.278165    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:22.897689 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:22.897653    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:27.760839 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:27.760803    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:27.863771 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:27.863735    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:32.570099 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:32.570066    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:32.679674 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:32.679638    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:35.063152 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:35.063118    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:35.187168 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:35.187139    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:36.647893 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:36.647858    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:36.754447 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:36.754420    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:36.860911 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:36.860867    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.546192 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.546159    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mztrk"]
Sep 16 18:47:37.548215 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.548197    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk"
Sep 16 18:47:37.550773 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.550752    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4zgdm\""
Sep 16 18:47:37.556232 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.556207    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24b7b\" (UniqueName: \"kubernetes.io/projected/e7a59102-5806-4837-8a6a-8d651ff4d2f6-kube-api-access-24b7b\") pod \"authorino-7bcfd8f9cd-mztrk\" (UID: \"e7a59102-5806-4837-8a6a-8d651ff4d2f6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk"
Sep 16 18:47:37.557390 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.557354    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mztrk"]
Sep 16 18:47:37.584943 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.584916    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.657233 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.657190    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24b7b\" (UniqueName: \"kubernetes.io/projected/e7a59102-5806-4837-8a6a-8d651ff4d2f6-kube-api-access-24b7b\") pod \"authorino-7bcfd8f9cd-mztrk\" (UID: \"e7a59102-5806-4837-8a6a-8d651ff4d2f6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk"
Sep 16 18:47:37.669772 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.669729    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24b7b\" (UniqueName: \"kubernetes.io/projected/e7a59102-5806-4837-8a6a-8d651ff4d2f6-kube-api-access-24b7b\") pod \"authorino-7bcfd8f9cd-mztrk\" (UID: \"e7a59102-5806-4837-8a6a-8d651ff4d2f6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk"
Sep 16 18:47:37.692235 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.692203    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.699633 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.699609    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-dknsv"]
Sep 16 18:47:37.701633 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.701619    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dknsv"
Sep 16 18:47:37.711999 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.711976    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dknsv"]
Sep 16 18:47:37.758015 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.757977    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4fzh\" (UniqueName: \"kubernetes.io/projected/d0fb1209-4696-4227-809d-5a6bca14bb9f-kube-api-access-s4fzh\") pod \"authorino-59c974bd48-dknsv\" (UID: \"d0fb1209-4696-4227-809d-5a6bca14bb9f\") " pod="kuadrant-system/authorino-59c974bd48-dknsv"
Sep 16 18:47:37.857569 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.857487    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk"
Sep 16 18:47:37.858738 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.858711    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4fzh\" (UniqueName: \"kubernetes.io/projected/d0fb1209-4696-4227-809d-5a6bca14bb9f-kube-api-access-s4fzh\") pod \"authorino-59c974bd48-dknsv\" (UID: \"d0fb1209-4696-4227-809d-5a6bca14bb9f\") " pod="kuadrant-system/authorino-59c974bd48-dknsv"
Sep 16 18:47:37.867893 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:37.867852    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4fzh\" (UniqueName: \"kubernetes.io/projected/d0fb1209-4696-4227-809d-5a6bca14bb9f-kube-api-access-s4fzh\") pod \"authorino-59c974bd48-dknsv\" (UID: \"d0fb1209-4696-4227-809d-5a6bca14bb9f\") " pod="kuadrant-system/authorino-59c974bd48-dknsv"
Sep 16 18:47:38.010330 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:38.010301    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dknsv"
Sep 16 18:47:38.125275 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:38.125203    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mztrk"]
Sep 16 18:47:38.128418 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:47:38.128393    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a59102_5806_4837_8a6a_8d651ff4d2f6.slice/crio-aa57da44fa9a3929398e08a5e14f8bb8652faef42ef1bdfabef11c0afbd80415 WatchSource:0}: Error finding container aa57da44fa9a3929398e08a5e14f8bb8652faef42ef1bdfabef11c0afbd80415: Status 404 returned error can't find the container with id aa57da44fa9a3929398e08a5e14f8bb8652faef42ef1bdfabef11c0afbd80415
Sep 16 18:47:38.280255 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:38.280221    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dknsv"]
Sep 16 18:47:38.282923 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:47:38.282898    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0fb1209_4696_4227_809d_5a6bca14bb9f.slice/crio-cd5020abfc4d7a370dd7b9ecc0617cb7bacdd58b8a026e0406411b2f98d683a6 WatchSource:0}: Error finding container cd5020abfc4d7a370dd7b9ecc0617cb7bacdd58b8a026e0406411b2f98d683a6: Status 404 returned error can't find the container with id cd5020abfc4d7a370dd7b9ecc0617cb7bacdd58b8a026e0406411b2f98d683a6
Sep 16 18:47:39.015866 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:39.015826    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dknsv" event={"ID":"d0fb1209-4696-4227-809d-5a6bca14bb9f","Type":"ContainerStarted","Data":"cd5020abfc4d7a370dd7b9ecc0617cb7bacdd58b8a026e0406411b2f98d683a6"}
Sep 16 18:47:39.017433 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:39.017391    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk" event={"ID":"e7a59102-5806-4837-8a6a-8d651ff4d2f6","Type":"ContainerStarted","Data":"aa57da44fa9a3929398e08a5e14f8bb8652faef42ef1bdfabef11c0afbd80415"}
Sep 16 18:47:41.028661 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:41.028621    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dknsv" event={"ID":"d0fb1209-4696-4227-809d-5a6bca14bb9f","Type":"ContainerStarted","Data":"8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79"}
Sep 16 18:47:41.030207 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:41.030185    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk" event={"ID":"e7a59102-5806-4837-8a6a-8d651ff4d2f6","Type":"ContainerStarted","Data":"35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629"}
Sep 16 18:47:41.046079 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:41.046035    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-dknsv" podStartSLOduration=2.318049588 podStartE2EDuration="4.046019764s" totalImagesPullingTime="1.727970176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:41.045145789 +0000 UTC m=+476.569604623" watchObservedRunningTime="2026-09-16 18:47:41.046019764 +0000 UTC m=+476.570478598"
Sep 16 18:47:41.069109 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:41.069041    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk" podStartSLOduration=2.182290198 podStartE2EDuration="4.069019692s" totalImagesPullingTime="1.886729494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:41.067090806 +0000 UTC m=+476.591549641" watchObservedRunningTime="2026-09-16 18:47:41.069019692 +0000 UTC m=+476.593478527"
Sep 16 18:47:41.084368 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:41.084335    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mztrk"]
Sep 16 18:47:43.038765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:43.038730    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk" podUID="e7a59102-5806-4837-8a6a-8d651ff4d2f6" containerName="authorino" containerID="cri-o://35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629" gracePeriod=30
Sep 16 18:47:43.281550 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:43.281523    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk"
Sep 16 18:47:43.300562 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:43.300495    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24b7b\" (UniqueName: \"kubernetes.io/projected/e7a59102-5806-4837-8a6a-8d651ff4d2f6-kube-api-access-24b7b\") pod \"e7a59102-5806-4837-8a6a-8d651ff4d2f6\" (UID: \"e7a59102-5806-4837-8a6a-8d651ff4d2f6\") "
Sep 16 18:47:43.303461 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:43.303432    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a59102-5806-4837-8a6a-8d651ff4d2f6-kube-api-access-24b7b" (OuterVolumeSpecName: "kube-api-access-24b7b") pod "e7a59102-5806-4837-8a6a-8d651ff4d2f6" (UID: "e7a59102-5806-4837-8a6a-8d651ff4d2f6"). InnerVolumeSpecName "kube-api-access-24b7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:43.401521 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:43.401485    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24b7b\" (UniqueName: \"kubernetes.io/projected/e7a59102-5806-4837-8a6a-8d651ff4d2f6-kube-api-access-24b7b\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:47:44.043969 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:44.043933    2590 generic.go:356] "Generic (PLEG): container finished" podID="e7a59102-5806-4837-8a6a-8d651ff4d2f6" containerID="35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629" exitCode=0
Sep 16 18:47:44.043969 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:44.043973    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk"
Sep 16 18:47:44.044488 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:44.044012    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk" event={"ID":"e7a59102-5806-4837-8a6a-8d651ff4d2f6","Type":"ContainerDied","Data":"35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629"}
Sep 16 18:47:44.044488 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:44.044054    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mztrk" event={"ID":"e7a59102-5806-4837-8a6a-8d651ff4d2f6","Type":"ContainerDied","Data":"aa57da44fa9a3929398e08a5e14f8bb8652faef42ef1bdfabef11c0afbd80415"}
Sep 16 18:47:44.044488 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:44.044085    2590 scope.go:117] "RemoveContainer" containerID="35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629"
Sep 16 18:47:44.053476 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:44.053457    2590 scope.go:117] "RemoveContainer" containerID="35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629"
Sep 16 18:47:44.053778 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:47:44.053757    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629\": container with ID starting with 35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629 not found: ID does not exist" containerID="35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629"
Sep 16 18:47:44.053828 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:44.053788    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629"} err="failed to get container status \"35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629\": rpc error: code = NotFound desc = could not find container \"35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629\": container with ID starting with 35dd8d91e426a7f50f61bed85171ee8a9fdafcbee5dcec4b11dbc0ae129e6629 not found: ID does not exist"
Sep 16 18:47:44.065108 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:44.065081    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mztrk"]
Sep 16 18:47:44.068269 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:44.068246    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mztrk"]
Sep 16 18:47:45.024552 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:45.024520    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a59102-5806-4837-8a6a-8d651ff4d2f6" path="/var/lib/kubelet/pods/e7a59102-5806-4837-8a6a-8d651ff4d2f6/volumes"
Sep 16 18:47:55.914526 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:55.914485    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:56.395156 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:56.395123    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:59.152005 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.151974    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-sh65m"]
Sep 16 18:47:59.152361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.152223    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a59102-5806-4837-8a6a-8d651ff4d2f6" containerName="authorino"
Sep 16 18:47:59.152361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.152233    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a59102-5806-4837-8a6a-8d651ff4d2f6" containerName="authorino"
Sep 16 18:47:59.152361 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.152298    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a59102-5806-4837-8a6a-8d651ff4d2f6" containerName="authorino"
Sep 16 18:47:59.154690 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.154672    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-sh65m"
Sep 16 18:47:59.157132 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.157095    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 18:47:59.164302 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.164281    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-sh65m"]
Sep 16 18:47:59.233789 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.233759    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4789d\" (UniqueName: \"kubernetes.io/projected/a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb-kube-api-access-4789d\") pod \"authorino-6d9898bd99-sh65m\" (UID: \"a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb\") " pod="kuadrant-system/authorino-6d9898bd99-sh65m"
Sep 16 18:47:59.233789 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.233799    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb-tls-cert\") pod \"authorino-6d9898bd99-sh65m\" (UID: \"a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb\") " pod="kuadrant-system/authorino-6d9898bd99-sh65m"
Sep 16 18:47:59.334386 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.334350    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4789d\" (UniqueName: \"kubernetes.io/projected/a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb-kube-api-access-4789d\") pod \"authorino-6d9898bd99-sh65m\" (UID: \"a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb\") " pod="kuadrant-system/authorino-6d9898bd99-sh65m"
Sep 16 18:47:59.334531 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.334396    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb-tls-cert\") pod \"authorino-6d9898bd99-sh65m\" (UID: \"a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb\") " pod="kuadrant-system/authorino-6d9898bd99-sh65m"
Sep 16 18:47:59.336979 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.336951    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb-tls-cert\") pod \"authorino-6d9898bd99-sh65m\" (UID: \"a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb\") " pod="kuadrant-system/authorino-6d9898bd99-sh65m"
Sep 16 18:47:59.343392 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.343369    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4789d\" (UniqueName: \"kubernetes.io/projected/a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb-kube-api-access-4789d\") pod \"authorino-6d9898bd99-sh65m\" (UID: \"a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb\") " pod="kuadrant-system/authorino-6d9898bd99-sh65m"
Sep 16 18:47:59.463335 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.463242    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-sh65m"
Sep 16 18:47:59.731679 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:47:59.731569    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-sh65m"]
Sep 16 18:47:59.734547 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:47:59.734519    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda25c6e13_0dc9_47aa_a060_9cf2bb0e1ecb.slice/crio-7f2b53a320813b75c568083ef221068ed83c522f35489b552e641fd852445088 WatchSource:0}: Error finding container 7f2b53a320813b75c568083ef221068ed83c522f35489b552e641fd852445088: Status 404 returned error can't find the container with id 7f2b53a320813b75c568083ef221068ed83c522f35489b552e641fd852445088
Sep 16 18:48:00.109492 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:00.109453    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-sh65m" event={"ID":"a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb","Type":"ContainerStarted","Data":"7f2b53a320813b75c568083ef221068ed83c522f35489b552e641fd852445088"}
Sep 16 18:48:01.114653 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:01.114603    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-sh65m" event={"ID":"a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb","Type":"ContainerStarted","Data":"8c73728feb9cf1920b9726191724e8fff831296a5ce0ecae748aeabe352bea08"}
Sep 16 18:48:01.134549 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:01.134496    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-sh65m" podStartSLOduration=1.741315856 podStartE2EDuration="2.134480464s" totalImagesPullingTime="393.164608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:01.133968793 +0000 UTC m=+496.658427642" watchObservedRunningTime="2026-09-16 18:48:01.134480464 +0000 UTC m=+496.658939298"
Sep 16 18:48:01.171177 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:01.171141    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dknsv"]
Sep 16 18:48:01.171353 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:01.171332    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-dknsv" podUID="d0fb1209-4696-4227-809d-5a6bca14bb9f" containerName="authorino" containerID="cri-o://8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79" gracePeriod=30
Sep 16 18:48:01.410757 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:01.410731    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dknsv"
Sep 16 18:48:01.552970 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:01.552932    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4fzh\" (UniqueName: \"kubernetes.io/projected/d0fb1209-4696-4227-809d-5a6bca14bb9f-kube-api-access-s4fzh\") pod \"d0fb1209-4696-4227-809d-5a6bca14bb9f\" (UID: \"d0fb1209-4696-4227-809d-5a6bca14bb9f\") "
Sep 16 18:48:01.555140 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:01.555116    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fb1209-4696-4227-809d-5a6bca14bb9f-kube-api-access-s4fzh" (OuterVolumeSpecName: "kube-api-access-s4fzh") pod "d0fb1209-4696-4227-809d-5a6bca14bb9f" (UID: "d0fb1209-4696-4227-809d-5a6bca14bb9f"). InnerVolumeSpecName "kube-api-access-s4fzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:01.654590 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:01.654509    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4fzh\" (UniqueName: \"kubernetes.io/projected/d0fb1209-4696-4227-809d-5a6bca14bb9f-kube-api-access-s4fzh\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:48:02.120399 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:02.120364    2590 generic.go:356] "Generic (PLEG): container finished" podID="d0fb1209-4696-4227-809d-5a6bca14bb9f" containerID="8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79" exitCode=0
Sep 16 18:48:02.120843 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:02.120414    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dknsv"
Sep 16 18:48:02.120843 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:02.120464    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dknsv" event={"ID":"d0fb1209-4696-4227-809d-5a6bca14bb9f","Type":"ContainerDied","Data":"8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79"}
Sep 16 18:48:02.120843 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:02.120513    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dknsv" event={"ID":"d0fb1209-4696-4227-809d-5a6bca14bb9f","Type":"ContainerDied","Data":"cd5020abfc4d7a370dd7b9ecc0617cb7bacdd58b8a026e0406411b2f98d683a6"}
Sep 16 18:48:02.120843 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:02.120539    2590 scope.go:117] "RemoveContainer" containerID="8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79"
Sep 16 18:48:02.128674 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:02.128657    2590 scope.go:117] "RemoveContainer" containerID="8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79"
Sep 16 18:48:02.128945 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:48:02.128927    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79\": container with ID starting with 8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79 not found: ID does not exist" containerID="8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79"
Sep 16 18:48:02.129000 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:02.128954    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79"} err="failed to get container status \"8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79\": rpc error: code = NotFound desc = could not find container \"8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79\": container with ID starting with 8016d482071912c78a061bbb6b21b234fa4b36969c74257c188006a1c58bff79 not found: ID does not exist"
Sep 16 18:48:02.144195 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:02.144159    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dknsv"]
Sep 16 18:48:02.147346 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:02.147325    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dknsv"]
Sep 16 18:48:03.024856 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:03.024822    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0fb1209-4696-4227-809d-5a6bca14bb9f" path="/var/lib/kubelet/pods/d0fb1209-4696-4227-809d-5a6bca14bb9f/volumes"
Sep 16 18:48:04.648597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:04.648564    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:04.747148 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:04.747111    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.320008 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:28.319972    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:33.900290 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:33.900245    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:33.976020 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:33.975980    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:34.107362 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:34.107314    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:34.209903 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:34.209779    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:34.972783 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:34.972751    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:35.107463 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:48:35.107411    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:06.977685 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:49:06.977640    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:07.091449 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:49:07.091403    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:56.386349 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:49:56.386314    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:56.495681 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:49:56.495646    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:56.592828 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:49:56.592790    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:56.693452 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:49:56.693369    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:56.922806 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:49:56.922773    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:57.593075 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:49:57.593042    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:57.659004 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:49:57.658973    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:14.216189 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.216103    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qf4tq"]
Sep 16 18:50:14.218437 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.216365    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fb1209-4696-4227-809d-5a6bca14bb9f" containerName="authorino"
Sep 16 18:50:14.218437 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.216375    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fb1209-4696-4227-809d-5a6bca14bb9f" containerName="authorino"
Sep 16 18:50:14.218437 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.216451    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fb1209-4696-4227-809d-5a6bca14bb9f" containerName="authorino"
Sep 16 18:50:14.219306 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.219291    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qf4tq"
Sep 16 18:50:14.221811 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.221787    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:50:14.221956 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.221831    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:50:14.221956 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.221839    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rqbgn\""
Sep 16 18:50:14.222518 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.222502    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:50:14.222631 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.222534    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:50:14.228612 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.228593    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qf4tq"]
Sep 16 18:50:14.344112 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.344075    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9lf\" (UniqueName: \"kubernetes.io/projected/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-kube-api-access-rs9lf\") pod \"s3-init-qf4tq\" (UID: \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\") " pod="kserve/s3-init-qf4tq"
Sep 16 18:50:14.344112 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.344114    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-model-cache\") pod \"s3-init-qf4tq\" (UID: \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\") " pod="kserve/s3-init-qf4tq"
Sep 16 18:50:14.445516 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.445471    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9lf\" (UniqueName: \"kubernetes.io/projected/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-kube-api-access-rs9lf\") pod \"s3-init-qf4tq\" (UID: \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\") " pod="kserve/s3-init-qf4tq"
Sep 16 18:50:14.445516 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.445520    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-model-cache\") pod \"s3-init-qf4tq\" (UID: \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\") " pod="kserve/s3-init-qf4tq"
Sep 16 18:50:14.445970 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.445949    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-model-cache\") pod \"s3-init-qf4tq\" (UID: \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\") " pod="kserve/s3-init-qf4tq"
Sep 16 18:50:14.459244 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.459215    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9lf\" (UniqueName: \"kubernetes.io/projected/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-kube-api-access-rs9lf\") pod \"s3-init-qf4tq\" (UID: \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\") " pod="kserve/s3-init-qf4tq"
Sep 16 18:50:14.528918 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.528869    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qf4tq"
Sep 16 18:50:14.815751 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:14.815670    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qf4tq"]
Sep 16 18:50:14.819296 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:50:14.819252    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode464a8d1_7497_4f4e_b1e2_2ec875216a5e.slice/crio-f33cdfc0ebbec12b6e649b904c5c798f615a82675dd6c1281121d0531cb4ad36 WatchSource:0}: Error finding container f33cdfc0ebbec12b6e649b904c5c798f615a82675dd6c1281121d0531cb4ad36: Status 404 returned error can't find the container with id f33cdfc0ebbec12b6e649b904c5c798f615a82675dd6c1281121d0531cb4ad36
Sep 16 18:50:15.678621 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:15.678581    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf4tq" event={"ID":"e464a8d1-7497-4f4e-b1e2-2ec875216a5e","Type":"ContainerStarted","Data":"f33cdfc0ebbec12b6e649b904c5c798f615a82675dd6c1281121d0531cb4ad36"}
Sep 16 18:50:19.699983 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:19.699945    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf4tq" event={"ID":"e464a8d1-7497-4f4e-b1e2-2ec875216a5e","Type":"ContainerStarted","Data":"647fabb4fd694d7fec213af97015b0bd340b180b84cb421398536cdc82cbfcc0"}
Sep 16 18:50:20.705654 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:20.705559    2590 generic.go:356] "Generic (PLEG): container finished" podID="e464a8d1-7497-4f4e-b1e2-2ec875216a5e" containerID="647fabb4fd694d7fec213af97015b0bd340b180b84cb421398536cdc82cbfcc0" exitCode=0
Sep 16 18:50:20.705654 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:20.705616    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf4tq" event={"ID":"e464a8d1-7497-4f4e-b1e2-2ec875216a5e","Type":"ContainerDied","Data":"647fabb4fd694d7fec213af97015b0bd340b180b84cb421398536cdc82cbfcc0"}
Sep 16 18:50:25.732460 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:25.732423    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf4tq" event={"ID":"e464a8d1-7497-4f4e-b1e2-2ec875216a5e","Type":"ContainerStarted","Data":"dd43ae21fab92de67b79c3d2c1f93af77bc3f4d799ff8fd28b40a132d69b2fe3"}
Sep 16 18:50:25.763765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:25.763714    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-qf4tq" podStartSLOduration=1.765345827 podStartE2EDuration="11.763699102s" totalImagesPullingTime="8.681396186s" totalInitContainerRuntime="1.316957089s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:25.763467777 +0000 UTC m=+641.287926614" watchObservedRunningTime="2026-09-16 18:50:25.763699102 +0000 UTC m=+641.288157935"
Sep 16 18:50:54.349702 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:50:54.349666    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:23.057346 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:23.057290    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:23.157679 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:23.157640    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:30.003296 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:30.003260    2590 generic.go:356] "Generic (PLEG): container finished" podID="e464a8d1-7497-4f4e-b1e2-2ec875216a5e" containerID="dd43ae21fab92de67b79c3d2c1f93af77bc3f4d799ff8fd28b40a132d69b2fe3" exitCode=0
Sep 16 18:51:30.003689 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:30.003334    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf4tq" event={"ID":"e464a8d1-7497-4f4e-b1e2-2ec875216a5e","Type":"ContainerDied","Data":"dd43ae21fab92de67b79c3d2c1f93af77bc3f4d799ff8fd28b40a132d69b2fe3"}
Sep 16 18:51:31.132528 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:31.132505    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qf4tq"
Sep 16 18:51:31.181227 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:31.181203    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs9lf\" (UniqueName: \"kubernetes.io/projected/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-kube-api-access-rs9lf\") pod \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\" (UID: \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\") "
Sep 16 18:51:31.181378 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:31.181288    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-model-cache\") pod \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\" (UID: \"e464a8d1-7497-4f4e-b1e2-2ec875216a5e\") "
Sep 16 18:51:31.183514 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:31.183485    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-kube-api-access-rs9lf" (OuterVolumeSpecName: "kube-api-access-rs9lf") pod "e464a8d1-7497-4f4e-b1e2-2ec875216a5e" (UID: "e464a8d1-7497-4f4e-b1e2-2ec875216a5e"). InnerVolumeSpecName "kube-api-access-rs9lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:31.200537 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:31.200505    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e464a8d1-7497-4f4e-b1e2-2ec875216a5e" (UID: "e464a8d1-7497-4f4e-b1e2-2ec875216a5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:31.282573 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:31.282543    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9lf\" (UniqueName: \"kubernetes.io/projected/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-kube-api-access-rs9lf\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:51:31.282573 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:31.282573    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e464a8d1-7497-4f4e-b1e2-2ec875216a5e-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:51:32.013569 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:32.013541    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf4tq" event={"ID":"e464a8d1-7497-4f4e-b1e2-2ec875216a5e","Type":"ContainerDied","Data":"f33cdfc0ebbec12b6e649b904c5c798f615a82675dd6c1281121d0531cb4ad36"}
Sep 16 18:51:32.013569 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:32.013572    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qf4tq"
Sep 16 18:51:32.013569 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:32.013576    2590 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f33cdfc0ebbec12b6e649b904c5c798f615a82675dd6c1281121d0531cb4ad36"
Sep 16 18:51:32.514644 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:32.514615    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:32.639117 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:32.639079    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:45.895036 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:45.894994    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:45.991617 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:51:45.991584    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:07.687166 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.687133    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"]
Sep 16 18:52:07.687612 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.687427    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e464a8d1-7497-4f4e-b1e2-2ec875216a5e" containerName="download-hf-model"
Sep 16 18:52:07.687612 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.687438    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="e464a8d1-7497-4f4e-b1e2-2ec875216a5e" containerName="download-hf-model"
Sep 16 18:52:07.687612 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.687447    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e464a8d1-7497-4f4e-b1e2-2ec875216a5e" containerName="s3-init"
Sep 16 18:52:07.687612 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.687452    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="e464a8d1-7497-4f4e-b1e2-2ec875216a5e" containerName="s3-init"
Sep 16 18:52:07.687612 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.687505    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="e464a8d1-7497-4f4e-b1e2-2ec875216a5e" containerName="s3-init"
Sep 16 18:52:07.690349 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.690329    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.693077 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.693054    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:52:07.693422 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.693402    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 18:52:07.693516 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.693407    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 18:52:07.693779 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.693754    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-vf57r\""
Sep 16 18:52:07.693779 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.693765    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 18:52:07.694547 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.694530    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 18:52:07.707200 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.707175    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"]
Sep 16 18:52:07.710980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.710955    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:07.777528 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.777491    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.777528 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.777528    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.777778 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.777571    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzk74\" (UniqueName: \"kubernetes.io/projected/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kube-api-access-jzk74\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.777778 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.777588    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.777778 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.777605    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.777778 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.777631    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.777778 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.777668    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.777778 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.777686    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879052 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879017    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzk74\" (UniqueName: \"kubernetes.io/projected/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kube-api-access-jzk74\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879052 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879053    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879304 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879072    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879304 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879196    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879304 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879261    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879304 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879282    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879498 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879333    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879498 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879355    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879498 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879479    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879630 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879553    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879739 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879716    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879800 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879757    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.879800 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.879777    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.881685 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.881667    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.881980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.881964    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:07.887536 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:07.887517    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzk74\" (UniqueName: \"kubernetes.io/projected/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kube-api-access-jzk74\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:08.000224 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:08.000123    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:08.283024 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:08.282988    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"]
Sep 16 18:52:08.286723 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:52:08.286690    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290cff61_9cd7_4a93_b75f_a3ac5b6b2294.slice/crio-16b6bdedcd624c8f98fb8c05df8530408cab76ad956ee82fbd729064df42ab14 WatchSource:0}: Error finding container 16b6bdedcd624c8f98fb8c05df8530408cab76ad956ee82fbd729064df42ab14: Status 404 returned error can't find the container with id 16b6bdedcd624c8f98fb8c05df8530408cab76ad956ee82fbd729064df42ab14
Sep 16 18:52:09.166037 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:09.166004    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" event={"ID":"290cff61-9cd7-4a93-b75f-a3ac5b6b2294","Type":"ContainerStarted","Data":"bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d"}
Sep 16 18:52:09.166482 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:09.166055    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" event={"ID":"290cff61-9cd7-4a93-b75f-a3ac5b6b2294","Type":"ContainerStarted","Data":"16b6bdedcd624c8f98fb8c05df8530408cab76ad956ee82fbd729064df42ab14"}
Sep 16 18:52:11.175708 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:11.175621    2590 generic.go:356] "Generic (PLEG): container finished" podID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerID="bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d" exitCode=0
Sep 16 18:52:11.176103 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:11.175698    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" event={"ID":"290cff61-9cd7-4a93-b75f-a3ac5b6b2294","Type":"ContainerDied","Data":"bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d"}
Sep 16 18:52:11.177111 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:11.177088    2590 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:52:18.112670 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:18.112582    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:18.549301 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:18.549262    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:21.873476 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:21.873438    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:21.971552 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:21.971520    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:24.254187 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:24.254154    2590 generic.go:356] "Generic (PLEG): container finished" podID="a9490cec-b84b-4cc7-aefa-946d4e5646fb" containerID="ae6b01464eaae175b3dffa1c2f51464cd6d0574e77c38e126c6edd1607bf5853" exitCode=0
Sep 16 18:52:24.254643 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:24.254231    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" event={"ID":"a9490cec-b84b-4cc7-aefa-946d4e5646fb","Type":"ContainerDied","Data":"ae6b01464eaae175b3dffa1c2f51464cd6d0574e77c38e126c6edd1607bf5853"}
Sep 16 18:52:24.254690 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:24.254664    2590 scope.go:117] "RemoveContainer" containerID="ae6b01464eaae175b3dffa1c2f51464cd6d0574e77c38e126c6edd1607bf5853"
Sep 16 18:52:24.555067 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:24.554930    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:52:24.555067 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:24.555047    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:52:24.555315 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:24.555108    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:52:25.019430 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:25.019346    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:25.261651 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:25.261612    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" event={"ID":"a9490cec-b84b-4cc7-aefa-946d4e5646fb","Type":"ContainerStarted","Data":"12c44f18e5758ef0ae95b9662e375f1f8d3c8580d2edb1827ff6e746337744b6"}
Sep 16 18:52:26.036427 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:26.036288    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:52:26.037560 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:26.037508    2590 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-blgj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 16 18:52:26.037709 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:26.037594    2590 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" podUID="a9490cec-b84b-4cc7-aefa-946d4e5646fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 16 18:52:27.029906 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:27.029256    2590 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-blgj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 16 18:52:27.029906 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:27.029320    2590 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" podUID="a9490cec-b84b-4cc7-aefa-946d4e5646fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 16 18:52:28.030908 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:28.028094    2590 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-blgj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 16 18:52:28.030908 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:28.028159    2590 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" podUID="a9490cec-b84b-4cc7-aefa-946d4e5646fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 16 18:52:28.728907 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:28.726994    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:52:29.023553 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:29.023058    2590 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-blgj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 16 18:52:29.023553 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:29.023117    2590 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" podUID="a9490cec-b84b-4cc7-aefa-946d4e5646fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 16 18:52:30.035941 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:30.031356    2590 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-blgj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Sep 16 18:52:30.035941 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:30.031426    2590 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4" podUID="a9490cec-b84b-4cc7-aefa-946d4e5646fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Sep 16 18:52:31.017687 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:31.017647    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:52:31.018782 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:31.018760    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-blgj4"
Sep 16 18:52:43.065504 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:43.065458    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:43.357007 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:43.356920    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" event={"ID":"290cff61-9cd7-4a93-b75f-a3ac5b6b2294","Type":"ContainerStarted","Data":"fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3"}
Sep 16 18:52:43.382524 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:43.382456    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podStartSLOduration=2.582878347 podStartE2EDuration="36.382435847s" totalImagesPullingTime="31.268513398s" totalInitContainerRuntime="2.531044102s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:43.381302206 +0000 UTC m=+778.905761063" watchObservedRunningTime="2026-09-16 18:52:43.382435847 +0000 UTC m=+778.906894682"
Sep 16 18:52:44.000621 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:44.000587    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:48.000430 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:48.000389    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:52:48.002044 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:48.002015    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:52:58.000718 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:58.000670    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:52:59.972136 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:52:59.972101    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:00.035118 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:00.035059    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:08.001605 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:08.001496    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:53:18.000939 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:18.000895    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:53:28.001307 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:28.001260    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:53:38.001125 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:38.001075    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:53:48.000540 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:48.000493    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:53:49.975240 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:49.970112    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:50.063354 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:50.063322    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:56.130525 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:56.130474    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:56.401089 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:56.400977    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:57.229065 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:57.229028    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:58.001140 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:53:58.001093    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:54:03.743422 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:03.743385    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:03.832495 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:03.832456    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:08.001097 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:08.001054    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:54:15.875763 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:15.875713    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:18.000809 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:18.000760    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:54:25.914892 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:25.914844    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:28.000985 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:28.000938    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:54:33.255098 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:33.255066    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:35.169284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:35.169252    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:35.265668 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:35.265634    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:38.011257 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:38.011180    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:54:38.019239 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:38.019213    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:54:42.375831 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:42.375783    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"]
Sep 16 18:54:42.379673 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:42.379626    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:57.398352 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:57.398292    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" containerID="cri-o://fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3" gracePeriod=45
Sep 16 18:54:58.011978 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.011941    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 16 18:54:58.141128 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.141101    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:54:58.164301 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164274    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-home\") pod \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") "
Sep 16 18:54:58.164470 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164316    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-dshm\") pod \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") "
Sep 16 18:54:58.164470 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164389    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-model-cache\") pod \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") "
Sep 16 18:54:58.164470 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164412    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tmp-dir\") pod \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") "
Sep 16 18:54:58.164470 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164440    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kserve-provision-location\") pod \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") "
Sep 16 18:54:58.164470 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164468    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzk74\" (UniqueName: \"kubernetes.io/projected/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kube-api-access-jzk74\") pod \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") "
Sep 16 18:54:58.164745 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164497    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-cabundle-cert\") pod \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") "
Sep 16 18:54:58.164745 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164522    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tls-certs\") pod \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\" (UID: \"290cff61-9cd7-4a93-b75f-a3ac5b6b2294\") "
Sep 16 18:54:58.164745 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164593    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-model-cache" (OuterVolumeSpecName: "model-cache") pod "290cff61-9cd7-4a93-b75f-a3ac5b6b2294" (UID: "290cff61-9cd7-4a93-b75f-a3ac5b6b2294"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:58.164961 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164785    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:54:58.164961 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.164929    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-home" (OuterVolumeSpecName: "home") pod "290cff61-9cd7-4a93-b75f-a3ac5b6b2294" (UID: "290cff61-9cd7-4a93-b75f-a3ac5b6b2294"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:58.165938 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.165193    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "290cff61-9cd7-4a93-b75f-a3ac5b6b2294" (UID: "290cff61-9cd7-4a93-b75f-a3ac5b6b2294"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:58.168206 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.168142    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "290cff61-9cd7-4a93-b75f-a3ac5b6b2294" (UID: "290cff61-9cd7-4a93-b75f-a3ac5b6b2294"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:58.168435 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.168399    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kube-api-access-jzk74" (OuterVolumeSpecName: "kube-api-access-jzk74") pod "290cff61-9cd7-4a93-b75f-a3ac5b6b2294" (UID: "290cff61-9cd7-4a93-b75f-a3ac5b6b2294"). InnerVolumeSpecName "kube-api-access-jzk74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:58.168659 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.168638    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-dshm" (OuterVolumeSpecName: "dshm") pod "290cff61-9cd7-4a93-b75f-a3ac5b6b2294" (UID: "290cff61-9cd7-4a93-b75f-a3ac5b6b2294"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:58.179260 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.179221    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "290cff61-9cd7-4a93-b75f-a3ac5b6b2294" (UID: "290cff61-9cd7-4a93-b75f-a3ac5b6b2294"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:58.192569 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.192539    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "290cff61-9cd7-4a93-b75f-a3ac5b6b2294" (UID: "290cff61-9cd7-4a93-b75f-a3ac5b6b2294"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:58.266138 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.266104    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:54:58.266138 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.266136    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:54:58.266138 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.266147    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzk74\" (UniqueName: \"kubernetes.io/projected/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-kube-api-access-jzk74\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:54:58.266366 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.266156    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:54:58.266366 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.266165    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:54:58.266366 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.266172    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:54:58.266366 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.266180    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290cff61-9cd7-4a93-b75f-a3ac5b6b2294-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:54:58.984538 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.984495    2590 generic.go:356] "Generic (PLEG): container finished" podID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerID="fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3" exitCode=0
Sep 16 18:54:58.985112 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.984546    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" event={"ID":"290cff61-9cd7-4a93-b75f-a3ac5b6b2294","Type":"ContainerDied","Data":"fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3"}
Sep 16 18:54:58.985112 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.984573    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"
Sep 16 18:54:58.985112 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.984633    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf" event={"ID":"290cff61-9cd7-4a93-b75f-a3ac5b6b2294","Type":"ContainerDied","Data":"16b6bdedcd624c8f98fb8c05df8530408cab76ad956ee82fbd729064df42ab14"}
Sep 16 18:54:58.985112 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.984665    2590 scope.go:117] "RemoveContainer" containerID="fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3"
Sep 16 18:54:58.994700 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:58.994676    2590 scope.go:117] "RemoveContainer" containerID="bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d"
Sep 16 18:54:59.002525 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:59.002505    2590 scope.go:117] "RemoveContainer" containerID="fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3"
Sep 16 18:54:59.002792 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:54:59.002773    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3\": container with ID starting with fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3 not found: ID does not exist" containerID="fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3"
Sep 16 18:54:59.002835 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:59.002800    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3"} err="failed to get container status \"fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3\": rpc error: code = NotFound desc = could not find container \"fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3\": container with ID starting with fe93baca0dde218e562afd9335506202a1fb255815bc94dd402722d0f7d52bc3 not found: ID does not exist"
Sep 16 18:54:59.002835 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:59.002818    2590 scope.go:117] "RemoveContainer" containerID="bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d"
Sep 16 18:54:59.003064 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:54:59.003049    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d\": container with ID starting with bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d not found: ID does not exist" containerID="bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d"
Sep 16 18:54:59.003103 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:59.003068    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d"} err="failed to get container status \"bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d\": rpc error: code = NotFound desc = could not find container \"bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d\": container with ID starting with bb62865362d8c2dd1f82ebbd0ae41d0a7a1c1a98a024553fcfd8ee38465c668d not found: ID does not exist"
Sep 16 18:54:59.009028 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:59.009007    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"]
Sep 16 18:54:59.016209 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:59.016188    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-579c87dd67j5jlf"]
Sep 16 18:54:59.024677 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:54:59.024655    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" path="/var/lib/kubelet/pods/290cff61-9cd7-4a93-b75f-a3ac5b6b2294/volumes"
Sep 16 18:55:02.388167 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:02.388136    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:02.481564 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:02.481531    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:04.900275 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:04.900240    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:05.102119 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.102081    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:05.200434 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.200355    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:05.225549 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.225517    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:55:05.225910 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.225872    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="storage-initializer"
Sep 16 18:55:05.225910 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.225911    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="storage-initializer"
Sep 16 18:55:05.226007 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.225926    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main"
Sep 16 18:55:05.226007 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.225931    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main"
Sep 16 18:55:05.226007 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.225985    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="290cff61-9cd7-4a93-b75f-a3ac5b6b2294" containerName="main"
Sep 16 18:55:05.233275 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.233249    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.236837 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.236806    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:55:05.237156 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.237140    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:55:05.237318 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.237305    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:55:05.237695 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.237669    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-w98nl\""
Sep 16 18:55:05.237821 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.237674    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-cx5b4\""
Sep 16 18:55:05.237821 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.237680    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:55:05.237821 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.237717    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:55:05.244063 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.244036    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:55:05.326787 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.326747    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62f8381-e460-49b5-8660-a2301ea31398-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.326997 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.326798    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.326997 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.326829    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.326997 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.326914    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.326997 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.326948    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.326997 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.326981    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62f8381-e460-49b5-8660-a2301ea31398-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.327205 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.327002    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cw5d\" (UniqueName: \"kubernetes.io/projected/b62f8381-e460-49b5-8660-a2301ea31398-kube-api-access-4cw5d\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.327205 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.327025    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.365734 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.365700    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:05.427706 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.427667    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62f8381-e460-49b5-8660-a2301ea31398-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.427706 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.427705    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cw5d\" (UniqueName: \"kubernetes.io/projected/b62f8381-e460-49b5-8660-a2301ea31398-kube-api-access-4cw5d\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.427998 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.427732    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.427998 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.427781    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62f8381-e460-49b5-8660-a2301ea31398-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.427998 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.427818    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.427998 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.427844    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.427998 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.427915    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.427998 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.427953    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.428306 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.428278    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.428368 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.428311    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.428368 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.428347    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.428368 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.428360    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.428473 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.428393    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62f8381-e460-49b5-8660-a2301ea31398-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.430184 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.430162    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.430419 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.430402    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62f8381-e460-49b5-8660-a2301ea31398-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.436496 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.436472    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cw5d\" (UniqueName: \"kubernetes.io/projected/b62f8381-e460-49b5-8660-a2301ea31398-kube-api-access-4cw5d\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.545003 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.544969    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:05.835662 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:05.835638    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:55:05.838359 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:55:05.838327    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62f8381_e460_49b5_8660_a2301ea31398.slice/crio-5fed2afc24e6b125c3896d20f6ddea635aa263b41bcec813d5c2470a539627fc WatchSource:0}: Error finding container 5fed2afc24e6b125c3896d20f6ddea635aa263b41bcec813d5c2470a539627fc: Status 404 returned error can't find the container with id 5fed2afc24e6b125c3896d20f6ddea635aa263b41bcec813d5c2470a539627fc
Sep 16 18:55:06.018963 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:06.018922    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b62f8381-e460-49b5-8660-a2301ea31398","Type":"ContainerStarted","Data":"ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7"}
Sep 16 18:55:06.019368 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:06.018998    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b62f8381-e460-49b5-8660-a2301ea31398","Type":"ContainerStarted","Data":"5fed2afc24e6b125c3896d20f6ddea635aa263b41bcec813d5c2470a539627fc"}
Sep 16 18:55:09.038569 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:09.038537    2590 generic.go:356] "Generic (PLEG): container finished" podID="b62f8381-e460-49b5-8660-a2301ea31398" containerID="ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7" exitCode=0
Sep 16 18:55:09.038950 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:09.038592    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b62f8381-e460-49b5-8660-a2301ea31398","Type":"ContainerDied","Data":"ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7"}
Sep 16 18:55:09.211574 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:09.211533    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:10.048226 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:10.048191    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b62f8381-e460-49b5-8660-a2301ea31398","Type":"ContainerStarted","Data":"c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7"}
Sep 16 18:55:10.070453 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:10.070397    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.7338047359999997 podStartE2EDuration="5.070382616s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.33657788s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:10.069315029 +0000 UTC m=+925.593773864" watchObservedRunningTime="2026-09-16 18:55:10.070382616 +0000 UTC m=+925.594841450"
Sep 16 18:55:10.546132 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:10.546098    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:14.190704 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:14.190662    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:15.408915 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:15.408854    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:15.467851 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:15.467819    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:15.545124 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:15.545092    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:15.546829 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:15.546799    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:55:25.141727 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.141689    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:25.251169 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.251130    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:25.364119 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.364074    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:25.480952 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.480918    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"]
Sep 16 18:55:25.484012 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.483988    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.491424 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.491392    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 18:55:25.491896 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.491839    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:55:25.491984 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.491916    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 18:55:25.491984 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.491967    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-lv5zd\""
Sep 16 18:55:25.505305 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.505278    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"]
Sep 16 18:55:25.528189 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.528143    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:25.545578 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.545535    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:55:25.604749 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.604707    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d48063-114f-47a1-8dd1-6957b55332f0-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.604749 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.604754    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.604980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.604793    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.604980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.604821    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.604980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.604838    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfx7z\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-kube-api-access-pfx7z\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.604980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.604864    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.604980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.604933    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.604980 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.604956    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d48063-114f-47a1-8dd1-6957b55332f0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.605173 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.604993    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.629610 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.629577    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:25.705863 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.705825    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.705863 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.705866    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706104 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.705903    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d48063-114f-47a1-8dd1-6957b55332f0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706104 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.705949    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706104 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.705972    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d48063-114f-47a1-8dd1-6957b55332f0-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706104 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.706005    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706311 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.706197    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706311 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.706250    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706311 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.706279    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfx7z\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-kube-api-access-pfx7z\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706459 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.706318    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706517 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.706494    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706608 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.706584    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.706918 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.706898    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d48063-114f-47a1-8dd1-6957b55332f0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.707052 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.707037    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.708761 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.708736    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d48063-114f-47a1-8dd1-6957b55332f0-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.709182 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.709159    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.713829 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.713799    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.714666 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.714642    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfx7z\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-kube-api-access-pfx7z\") pod \"router-gateway-1-openshift-default-6b46477c5b-6d7hh\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:25.795215 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:25.795180    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:26.074124 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:26.074101    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"]
Sep 16 18:55:26.076793 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:55:26.076759    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d48063_114f_47a1_8dd1_6957b55332f0.slice/crio-1a02efbe902dcb39a6bfd62a4afeac73467e78f25f872763629605b7d3b9b560 WatchSource:0}: Error finding container 1a02efbe902dcb39a6bfd62a4afeac73467e78f25f872763629605b7d3b9b560: Status 404 returned error can't find the container with id 1a02efbe902dcb39a6bfd62a4afeac73467e78f25f872763629605b7d3b9b560
Sep 16 18:55:26.079058 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:26.079027    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:55:26.079143 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:26.079105    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:55:26.079143 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:26.079133    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:55:26.120300 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:26.120271    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh" event={"ID":"66d48063-114f-47a1-8dd1-6957b55332f0","Type":"ContainerStarted","Data":"1a02efbe902dcb39a6bfd62a4afeac73467e78f25f872763629605b7d3b9b560"}
Sep 16 18:55:27.126597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:27.126561    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh" event={"ID":"66d48063-114f-47a1-8dd1-6957b55332f0","Type":"ContainerStarted","Data":"ca1ab12481149b1524817061d6023d48f4a26b64fb1b9ed6ef911008d28c8f9b"}
Sep 16 18:55:27.151622 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:27.151563    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh" podStartSLOduration=2.151546092 podStartE2EDuration="2.151546092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:27.14907424 +0000 UTC m=+942.673533117" watchObservedRunningTime="2026-09-16 18:55:27.151546092 +0000 UTC m=+942.676004926"
Sep 16 18:55:27.795986 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:27.795943    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:27.801429 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:27.801397    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:28.131347 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:28.131265    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:28.132301 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:28.132277    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:28.920660 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:28.920624    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:31.385380 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:31.385345    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:31.760284 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:31.760195    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:34.119119 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.119086    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:34.263636 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.263596    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:34.406530 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.406409    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:34.800636 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.800596    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:34.875336 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.875300    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"]
Sep 16 18:55:34.879220 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.879195    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.883343 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.883319    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-6s5kn\""
Sep 16 18:55:34.883761 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.883741    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:55:34.886505 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.886456    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 18:55:34.886612 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.886591    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 18:55:34.943279 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.943246    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:34.988176 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.988142    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl4l9\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-kube-api-access-wl4l9\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.988367 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.988183    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.988367 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.988231    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.988367 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.988280    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.988367 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.988348    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.988577 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.988393    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.988577 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.988430    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.988577 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.988542    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.988720 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.988606    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:34.991276 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:34.991252    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"]
Sep 16 18:55:35.089325 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089290    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl4l9\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-kube-api-access-wl4l9\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.089325 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089327    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.089581 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089354    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.089581 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089372    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.089581 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089389    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.089581 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089414    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.089581 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089447    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.089841 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089631    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.089841 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089678    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.089841 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089814    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.090015 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.089912    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.090065 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.090012    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.090213 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.090193    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.090303 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.090271    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.092296 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.092261    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.092386 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.092310    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.103383 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.103350    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl4l9\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-kube-api-access-wl4l9\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.104804 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.104782    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-gkj92\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.189968 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.189929    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:35.504055 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.503999    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"]
Sep 16 18:55:35.506688 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:55:35.506657    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9ddb99_b669_4cdd_bcf4_0fc7325a9191.slice/crio-3c1ff4b75fe987feda10b96daeb0f2da04f76f2a8c9cdf03c5b38b3f8203b052 WatchSource:0}: Error finding container 3c1ff4b75fe987feda10b96daeb0f2da04f76f2a8c9cdf03c5b38b3f8203b052: Status 404 returned error can't find the container with id 3c1ff4b75fe987feda10b96daeb0f2da04f76f2a8c9cdf03c5b38b3f8203b052
Sep 16 18:55:35.508810 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.508780    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:55:35.508891 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.508844    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:55:35.508939 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.508916    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:55:35.545569 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:35.545489    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:55:36.172491 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:36.172452    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92" event={"ID":"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191","Type":"ContainerStarted","Data":"cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc"}
Sep 16 18:55:36.172690 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:36.172503    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92" event={"ID":"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191","Type":"ContainerStarted","Data":"3c1ff4b75fe987feda10b96daeb0f2da04f76f2a8c9cdf03c5b38b3f8203b052"}
Sep 16 18:55:36.190585 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:36.190545    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:36.196059 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:36.196030    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:36.197957 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:36.197914    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92" podStartSLOduration=2.197897805 podStartE2EDuration="2.197897805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:36.196225288 +0000 UTC m=+951.720684123" watchObservedRunningTime="2026-09-16 18:55:36.197897805 +0000 UTC m=+951.722356629"
Sep 16 18:55:37.177135 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:37.177098    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:37.178158 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:37.178135    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:37.725941 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:37.725852    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:38.047600 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:38.047568    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:38.220251 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:38.220202    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:38.324827 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:38.324697    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:39.081829 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:39.081784    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"]
Sep 16 18:55:39.082263 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:39.082063    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh" podUID="66d48063-114f-47a1-8dd1-6957b55332f0" containerName="istio-proxy" containerID="cri-o://ca1ab12481149b1524817061d6023d48f4a26b64fb1b9ed6ef911008d28c8f9b" gracePeriod=30
Sep 16 18:55:39.179146 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:39.179113    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:41.197548 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.197517    2590 generic.go:356] "Generic (PLEG): container finished" podID="66d48063-114f-47a1-8dd1-6957b55332f0" containerID="ca1ab12481149b1524817061d6023d48f4a26b64fb1b9ed6ef911008d28c8f9b" exitCode=0
Sep 16 18:55:41.197941 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.197588    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh" event={"ID":"66d48063-114f-47a1-8dd1-6957b55332f0","Type":"ContainerDied","Data":"ca1ab12481149b1524817061d6023d48f4a26b64fb1b9ed6ef911008d28c8f9b"}
Sep 16 18:55:41.322895 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.322861    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:41.449514 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449462    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d48063-114f-47a1-8dd1-6957b55332f0-istiod-ca-cert\") pod \"66d48063-114f-47a1-8dd1-6957b55332f0\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") "
Sep 16 18:55:41.449514 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449521    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-certs\") pod \"66d48063-114f-47a1-8dd1-6957b55332f0\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") "
Sep 16 18:55:41.449772 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449569    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-data\") pod \"66d48063-114f-47a1-8dd1-6957b55332f0\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") "
Sep 16 18:55:41.449772 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449625    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-istio-token\") pod \"66d48063-114f-47a1-8dd1-6957b55332f0\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") "
Sep 16 18:55:41.449772 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449653    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-credential-socket\") pod \"66d48063-114f-47a1-8dd1-6957b55332f0\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") "
Sep 16 18:55:41.449772 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449739    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-socket\") pod \"66d48063-114f-47a1-8dd1-6957b55332f0\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") "
Sep 16 18:55:41.449772 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449771    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d48063-114f-47a1-8dd1-6957b55332f0-istio-podinfo\") pod \"66d48063-114f-47a1-8dd1-6957b55332f0\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") "
Sep 16 18:55:41.450062 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449802    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-envoy\") pod \"66d48063-114f-47a1-8dd1-6957b55332f0\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") "
Sep 16 18:55:41.450062 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449835    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfx7z\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-kube-api-access-pfx7z\") pod \"66d48063-114f-47a1-8dd1-6957b55332f0\" (UID: \"66d48063-114f-47a1-8dd1-6957b55332f0\") "
Sep 16 18:55:41.450062 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449914    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d48063-114f-47a1-8dd1-6957b55332f0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66d48063-114f-47a1-8dd1-6957b55332f0" (UID: "66d48063-114f-47a1-8dd1-6957b55332f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:41.450062 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.449920    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-data" (OuterVolumeSpecName: "istio-data") pod "66d48063-114f-47a1-8dd1-6957b55332f0" (UID: "66d48063-114f-47a1-8dd1-6957b55332f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:41.450370 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.450106    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66d48063-114f-47a1-8dd1-6957b55332f0" (UID: "66d48063-114f-47a1-8dd1-6957b55332f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:41.450370 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.450147    2590 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-data\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.450370 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.450168    2590 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d48063-114f-47a1-8dd1-6957b55332f0-istiod-ca-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.450370 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.450160    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66d48063-114f-47a1-8dd1-6957b55332f0" (UID: "66d48063-114f-47a1-8dd1-6957b55332f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:41.450370 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.450334    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66d48063-114f-47a1-8dd1-6957b55332f0" (UID: "66d48063-114f-47a1-8dd1-6957b55332f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:41.452762 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.452729    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66d48063-114f-47a1-8dd1-6957b55332f0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66d48063-114f-47a1-8dd1-6957b55332f0" (UID: "66d48063-114f-47a1-8dd1-6957b55332f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:55:41.452989 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.452776    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-kube-api-access-pfx7z" (OuterVolumeSpecName: "kube-api-access-pfx7z") pod "66d48063-114f-47a1-8dd1-6957b55332f0" (UID: "66d48063-114f-47a1-8dd1-6957b55332f0"). InnerVolumeSpecName "kube-api-access-pfx7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:41.452989 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.452842    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "66d48063-114f-47a1-8dd1-6957b55332f0" (UID: "66d48063-114f-47a1-8dd1-6957b55332f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:41.452989 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.452951    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66d48063-114f-47a1-8dd1-6957b55332f0" (UID: "66d48063-114f-47a1-8dd1-6957b55332f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:41.551626 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.551590    2590 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-istio-token\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.551626 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.551621    2590 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-credential-socket\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.551626 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.551633    2590 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-socket\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.551856 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.551642    2590 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d48063-114f-47a1-8dd1-6957b55332f0-istio-podinfo\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.551856 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.551651    2590 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-istio-envoy\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.551856 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.551660    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfx7z\" (UniqueName: \"kubernetes.io/projected/66d48063-114f-47a1-8dd1-6957b55332f0-kube-api-access-pfx7z\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.551856 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:41.551668    2590 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d48063-114f-47a1-8dd1-6957b55332f0-workload-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:42.204479 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:42.204448    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"
Sep 16 18:55:42.204997 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:42.204450    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh" event={"ID":"66d48063-114f-47a1-8dd1-6957b55332f0","Type":"ContainerDied","Data":"1a02efbe902dcb39a6bfd62a4afeac73467e78f25f872763629605b7d3b9b560"}
Sep 16 18:55:42.204997 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:42.204602    2590 scope.go:117] "RemoveContainer" containerID="ca1ab12481149b1524817061d6023d48f4a26b64fb1b9ed6ef911008d28c8f9b"
Sep 16 18:55:42.229268 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:42.229239    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"]
Sep 16 18:55:42.233002 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:42.232977    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6d7hh"]
Sep 16 18:55:43.026489 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:43.026455    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d48063-114f-47a1-8dd1-6957b55332f0" path="/var/lib/kubelet/pods/66d48063-114f-47a1-8dd1-6957b55332f0/volumes"
Sep 16 18:55:44.217466 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:44.217435    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:45.500829 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:45.500737    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:45.546346 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:45.546294    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:55:49.261479 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:49.261449    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:50.580086 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:50.580052    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:51.634863 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:51.634820    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:51.724893 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:51.724840    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:52.460293 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:52.460252    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:52.563257 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:52.563219    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:53.427336 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:53.427267    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:55.545630 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:55.545592    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:55:56.553545 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:56.553504    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"]
Sep 16 18:55:56.553975 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:56.553746    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92" podUID="0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" containerName="istio-proxy" containerID="cri-o://cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc" gracePeriod=30
Sep 16 18:55:56.651211 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:56.651160    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:58.800300 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.800274    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:58.913194 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913162    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-socket\") pod \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") "
Sep 16 18:55:58.913194 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913200    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-data\") pod \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") "
Sep 16 18:55:58.913432 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913238    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istiod-ca-cert\") pod \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") "
Sep 16 18:55:58.913432 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913282    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-envoy\") pod \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") "
Sep 16 18:55:58.913432 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913299    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl4l9\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-kube-api-access-wl4l9\") pod \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") "
Sep 16 18:55:58.913432 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913329    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-credential-socket\") pod \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") "
Sep 16 18:55:58.913432 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913369    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-podinfo\") pod \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") "
Sep 16 18:55:58.913678 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913437    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-certs\") pod \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") "
Sep 16 18:55:58.913678 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913461    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-token\") pod \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\" (UID: \"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191\") "
Sep 16 18:55:58.913678 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913470    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" (UID: "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:58.913678 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913552    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" (UID: "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:58.913678 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913646    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" (UID: "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:58.913961 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913676    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" (UID: "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:58.913961 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913741    2590 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:58.913961 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913762    2590 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-workload-socket\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:58.913961 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913774    2590 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-data\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:58.913961 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.913943    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" (UID: "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:58.915790 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.915767    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" (UID: "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:58.915932 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.915867    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" (UID: "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:58.915994 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.915957    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" (UID: "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:55:58.916200 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:58.916174    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-kube-api-access-wl4l9" (OuterVolumeSpecName: "kube-api-access-wl4l9") pod "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" (UID: "0f9ddb99-b669-4cdd-bcf4-0fc7325a9191"). InnerVolumeSpecName "kube-api-access-wl4l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:59.014318 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.014285    2590 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-token\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:59.014318 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.014313    2590 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istiod-ca-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:59.014318 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.014322    2590 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-envoy\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:59.014552 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.014331    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl4l9\" (UniqueName: \"kubernetes.io/projected/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-kube-api-access-wl4l9\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:59.014552 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.014341    2590 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-credential-socket\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:59.014552 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.014349    2590 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191-istio-podinfo\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:59.288788 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.288748    2590 generic.go:356] "Generic (PLEG): container finished" podID="0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" containerID="cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc" exitCode=0
Sep 16 18:55:59.289006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.288796    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92" event={"ID":"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191","Type":"ContainerDied","Data":"cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc"}
Sep 16 18:55:59.289006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.288818    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"
Sep 16 18:55:59.289006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.288893    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92" event={"ID":"0f9ddb99-b669-4cdd-bcf4-0fc7325a9191","Type":"ContainerDied","Data":"3c1ff4b75fe987feda10b96daeb0f2da04f76f2a8c9cdf03c5b38b3f8203b052"}
Sep 16 18:55:59.289006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.288911    2590 scope.go:117] "RemoveContainer" containerID="cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc"
Sep 16 18:55:59.299125 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.299098    2590 scope.go:117] "RemoveContainer" containerID="cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc"
Sep 16 18:55:59.299428 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:55:59.299410    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc\": container with ID starting with cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc not found: ID does not exist" containerID="cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc"
Sep 16 18:55:59.299484 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.299437    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc"} err="failed to get container status \"cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc\": rpc error: code = NotFound desc = could not find container \"cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc\": container with ID starting with cc286ec563b8b409611d29d921b58a9005e186f0c1460498f33ae85508c9e3bc not found: ID does not exist"
Sep 16 18:55:59.307174 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.307145    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"]
Sep 16 18:55:59.313824 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:55:59.313800    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-gkj92"]
Sep 16 18:56:01.025502 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:01.025469    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" path="/var/lib/kubelet/pods/0f9ddb99-b669-4cdd-bcf4-0fc7325a9191/volumes"
Sep 16 18:56:01.687664 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:01.687632    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:03.364870 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:03.364834    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:05.546233 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:05.546194    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:56:08.004928 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:08.004897    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:09.332260 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:09.332221    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:09.397002 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:09.396970    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:12.950929 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.950897    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"]
Sep 16 18:56:12.951390 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.951309    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" containerName="istio-proxy"
Sep 16 18:56:12.951390 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.951327    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" containerName="istio-proxy"
Sep 16 18:56:12.951390 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.951348    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d48063-114f-47a1-8dd1-6957b55332f0" containerName="istio-proxy"
Sep 16 18:56:12.951390 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.951356    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d48063-114f-47a1-8dd1-6957b55332f0" containerName="istio-proxy"
Sep 16 18:56:12.951597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.951429    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9ddb99-b669-4cdd-bcf4-0fc7325a9191" containerName="istio-proxy"
Sep 16 18:56:12.951597 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.951446    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d48063-114f-47a1-8dd1-6957b55332f0" containerName="istio-proxy"
Sep 16 18:56:12.954139 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.954117    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:12.956685 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.956663    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:56:12.956827 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.956812    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 18:56:12.956920 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.956850    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 18:56:12.956920 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.956901    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-z5rh6\""
Sep 16 18:56:12.957218 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.957202    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 18:56:12.957312 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.957240    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 18:56:12.969106 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.969074    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"]
Sep 16 18:56:12.977575 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:12.977549    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:13.041558 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.041525    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-model-cache\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.041725 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.041565    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-home\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.041725 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.041585    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tmp-dir\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.041725 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.041653    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-dshm\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.041725 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.041669    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tls-certs\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.041725 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.041688    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kserve-provision-location\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.041928 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.041743    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd6zw\" (UniqueName: \"kubernetes.io/projected/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kube-api-access-bd6zw\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.041928 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.041808    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-cabundle-cert\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.142681 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.142647    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-dshm\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.142681 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.142681    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tls-certs\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.142948 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.142702    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kserve-provision-location\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.142948 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.142723    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd6zw\" (UniqueName: \"kubernetes.io/projected/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kube-api-access-bd6zw\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.142948 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.142902    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-cabundle-cert\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.143107 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.142999    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-model-cache\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.143107 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.143055    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-home\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.143107 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.143091    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tmp-dir\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.143252 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.143177    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kserve-provision-location\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.143350 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.143330    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-home\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.143466 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.143443    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tmp-dir\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.143604 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.143585    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-model-cache\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.143639 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.143585    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-cabundle-cert\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.145681 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.145655    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-dshm\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.145872 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.145853    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tls-certs\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.152006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.151984    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd6zw\" (UniqueName: \"kubernetes.io/projected/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kube-api-access-bd6zw\") pod \"auth-enabled-test-kserve-7496c4dff4-j6pk7\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.225108 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.225017    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:13.266814 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.266776    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:13.548891 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:13.548853    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"]
Sep 16 18:56:13.551110 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:56:13.551073    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14e57a8_fb3b_4abb_86d3_9ae166ed141d.slice/crio-e91ccbf30c33ee49d6b776b167f1755ecc61bcc589f77a8f604b5d422f9241ad WatchSource:0}: Error finding container e91ccbf30c33ee49d6b776b167f1755ecc61bcc589f77a8f604b5d422f9241ad: Status 404 returned error can't find the container with id e91ccbf30c33ee49d6b776b167f1755ecc61bcc589f77a8f604b5d422f9241ad
Sep 16 18:56:14.362501 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:14.362465    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" event={"ID":"a14e57a8-fb3b-4abb-86d3-9ae166ed141d","Type":"ContainerStarted","Data":"bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2"}
Sep 16 18:56:14.362962 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:14.362526    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" event={"ID":"a14e57a8-fb3b-4abb-86d3-9ae166ed141d","Type":"ContainerStarted","Data":"e91ccbf30c33ee49d6b776b167f1755ecc61bcc589f77a8f604b5d422f9241ad"}
Sep 16 18:56:15.545804 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:15.545740    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:56:16.372977 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:16.372939    2590 generic.go:356] "Generic (PLEG): container finished" podID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerID="bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2" exitCode=0
Sep 16 18:56:16.373157 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:16.373014    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" event={"ID":"a14e57a8-fb3b-4abb-86d3-9ae166ed141d","Type":"ContainerDied","Data":"bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2"}
Sep 16 18:56:17.379824 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:17.379793    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" event={"ID":"a14e57a8-fb3b-4abb-86d3-9ae166ed141d","Type":"ContainerStarted","Data":"bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415"}
Sep 16 18:56:17.413067 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:17.413008    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podStartSLOduration=3.027822958 podStartE2EDuration="5.412991837s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.385168879s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:17.411162182 +0000 UTC m=+992.935621017" watchObservedRunningTime="2026-09-16 18:56:17.412991837 +0000 UTC m=+992.937450670"
Sep 16 18:56:18.267840 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:18.267805    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:23.267594 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:23.267561    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:56:23.269262 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:23.269225    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:56:25.545822 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:25.545761    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:56:28.151469 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:28.151438    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:28.214735 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:28.214696    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:33.267396 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:33.267347    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:56:35.545687 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:35.545639    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:56:36.521193 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:36.521155    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:43.268179 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:43.268122    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:56:45.546049 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:45.545998    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 16 18:56:53.267286 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:53.267243    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:56:55.555785 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:55.555747    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:56:55.563520 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:56:55.563489    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:00.805558 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:00.805487    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:00.846761 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:00.846721    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:57:00.870946 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:00.870903    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:03.268016 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:03.267972    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:57:13.267750 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:13.267701    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:57:15.870146 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:15.870085    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" containerID="cri-o://c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7" gracePeriod=45
Sep 16 18:57:16.616801 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.616769    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:16.672852 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.672770    2590 generic.go:356] "Generic (PLEG): container finished" podID="b62f8381-e460-49b5-8660-a2301ea31398" containerID="c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7" exitCode=0
Sep 16 18:57:16.672852 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.672818    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:16.673057 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.672852    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b62f8381-e460-49b5-8660-a2301ea31398","Type":"ContainerDied","Data":"c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7"}
Sep 16 18:57:16.673057 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.672928    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b62f8381-e460-49b5-8660-a2301ea31398","Type":"ContainerDied","Data":"5fed2afc24e6b125c3896d20f6ddea635aa263b41bcec813d5c2470a539627fc"}
Sep 16 18:57:16.673057 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.672959    2590 scope.go:117] "RemoveContainer" containerID="c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7"
Sep 16 18:57:16.681651 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.681631    2590 scope.go:117] "RemoveContainer" containerID="ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7"
Sep 16 18:57:16.689235 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.689210    2590 scope.go:117] "RemoveContainer" containerID="c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7"
Sep 16 18:57:16.689564 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:57:16.689545    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7\": container with ID starting with c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7 not found: ID does not exist" containerID="c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7"
Sep 16 18:57:16.689640 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.689578    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7"} err="failed to get container status \"c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7\": rpc error: code = NotFound desc = could not find container \"c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7\": container with ID starting with c212ba1f9df75a579999a8e392a00f7602b16bad269da1acafb9f99b226032b7 not found: ID does not exist"
Sep 16 18:57:16.689640 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.689607    2590 scope.go:117] "RemoveContainer" containerID="ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7"
Sep 16 18:57:16.689897 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:57:16.689857    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7\": container with ID starting with ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7 not found: ID does not exist" containerID="ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7"
Sep 16 18:57:16.689965 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.689913    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7"} err="failed to get container status \"ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7\": rpc error: code = NotFound desc = could not find container \"ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7\": container with ID starting with ecb7b2680f68b6b8f8a843ecccfe9086326a13b34fff454d010f2b240b5410d7 not found: ID does not exist"
Sep 16 18:57:16.712775 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.712750    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62f8381-e460-49b5-8660-a2301ea31398-cabundle-cert\") pod \"b62f8381-e460-49b5-8660-a2301ea31398\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") "
Sep 16 18:57:16.712910 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.712785    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62f8381-e460-49b5-8660-a2301ea31398-tls-certs\") pod \"b62f8381-e460-49b5-8660-a2301ea31398\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") "
Sep 16 18:57:16.712910 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.712819    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-tmp-dir\") pod \"b62f8381-e460-49b5-8660-a2301ea31398\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") "
Sep 16 18:57:16.712910 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.712849    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cw5d\" (UniqueName: \"kubernetes.io/projected/b62f8381-e460-49b5-8660-a2301ea31398-kube-api-access-4cw5d\") pod \"b62f8381-e460-49b5-8660-a2301ea31398\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") "
Sep 16 18:57:16.713070 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.712910    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-dshm\") pod \"b62f8381-e460-49b5-8660-a2301ea31398\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") "
Sep 16 18:57:16.713070 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.712942    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-home\") pod \"b62f8381-e460-49b5-8660-a2301ea31398\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") "
Sep 16 18:57:16.713070 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.712968    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-model-cache\") pod \"b62f8381-e460-49b5-8660-a2301ea31398\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") "
Sep 16 18:57:16.713070 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.713021    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-kserve-provision-location\") pod \"b62f8381-e460-49b5-8660-a2301ea31398\" (UID: \"b62f8381-e460-49b5-8660-a2301ea31398\") "
Sep 16 18:57:16.713327 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.713184    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62f8381-e460-49b5-8660-a2301ea31398-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b62f8381-e460-49b5-8660-a2301ea31398" (UID: "b62f8381-e460-49b5-8660-a2301ea31398"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:16.713382 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.713369    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62f8381-e460-49b5-8660-a2301ea31398-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:57:16.713522 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.713477    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-model-cache" (OuterVolumeSpecName: "model-cache") pod "b62f8381-e460-49b5-8660-a2301ea31398" (UID: "b62f8381-e460-49b5-8660-a2301ea31398"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:16.714058 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.714012    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-home" (OuterVolumeSpecName: "home") pod "b62f8381-e460-49b5-8660-a2301ea31398" (UID: "b62f8381-e460-49b5-8660-a2301ea31398"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:16.715603 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.715567    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62f8381-e460-49b5-8660-a2301ea31398-kube-api-access-4cw5d" (OuterVolumeSpecName: "kube-api-access-4cw5d") pod "b62f8381-e460-49b5-8660-a2301ea31398" (UID: "b62f8381-e460-49b5-8660-a2301ea31398"). InnerVolumeSpecName "kube-api-access-4cw5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:16.715603 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.715592    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62f8381-e460-49b5-8660-a2301ea31398-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b62f8381-e460-49b5-8660-a2301ea31398" (UID: "b62f8381-e460-49b5-8660-a2301ea31398"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:16.716145 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.716111    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-dshm" (OuterVolumeSpecName: "dshm") pod "b62f8381-e460-49b5-8660-a2301ea31398" (UID: "b62f8381-e460-49b5-8660-a2301ea31398"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:16.726459 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.726430    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b62f8381-e460-49b5-8660-a2301ea31398" (UID: "b62f8381-e460-49b5-8660-a2301ea31398"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:16.737587 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.737554    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b62f8381-e460-49b5-8660-a2301ea31398" (UID: "b62f8381-e460-49b5-8660-a2301ea31398"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:16.814444 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.814394    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:57:16.814444 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.814435    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:57:16.814444 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.814443    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:57:16.814444 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.814460    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:57:16.814720 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.814470    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62f8381-e460-49b5-8660-a2301ea31398-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:57:16.814720 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.814480    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b62f8381-e460-49b5-8660-a2301ea31398-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:57:16.814720 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.814490    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cw5d\" (UniqueName: \"kubernetes.io/projected/b62f8381-e460-49b5-8660-a2301ea31398-kube-api-access-4cw5d\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:57:16.998758 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:16.998725    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:57:17.002051 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:17.002022    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:57:17.025315 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:17.025278    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b62f8381-e460-49b5-8660-a2301ea31398" path="/var/lib/kubelet/pods/b62f8381-e460-49b5-8660-a2301ea31398/volumes"
Sep 16 18:57:17.556614 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:17.556571    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": context deadline exceeded"
Sep 16 18:57:23.267600 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:23.267557    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:57:33.267898 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:33.267849    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:57:43.268085 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:43.268038    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:57:50.921970 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:50.921933    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:51.023550 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:51.023507    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:53.268142 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:53.268093    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:57:58.102312 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:57:58.102279    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:01.701079 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:01.701043    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:01.959414 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:01.959326    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:03.267765 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:03.267717    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:58:05.657512 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:05.657477    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:05.720199 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:05.720149    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:08.188674 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:08.188633    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:13.278340 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:13.278307    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:58:13.286118 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:13.286099    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:58:17.556226 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:17.556176    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:18.692289 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:18.692259    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:18.731781 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:18.731751    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"]
Sep 16 18:58:18.777711 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:18.777676    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:33.754639 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:33.754571    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" containerID="cri-o://bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415" gracePeriod=45
Sep 16 18:58:34.279057 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.279010    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Sep 16 18:58:34.500028 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.500004    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:58:34.589527 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.589433    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tls-certs\") pod \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") "
Sep 16 18:58:34.589527 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.589482    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd6zw\" (UniqueName: \"kubernetes.io/projected/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kube-api-access-bd6zw\") pod \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") "
Sep 16 18:58:34.589527 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.589506    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-home\") pod \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") "
Sep 16 18:58:34.589817 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.589534    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-dshm\") pod \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") "
Sep 16 18:58:34.589817 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.589667    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-cabundle-cert\") pod \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") "
Sep 16 18:58:34.589817 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.589772    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-model-cache\") pod \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") "
Sep 16 18:58:34.589817 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.589811    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kserve-provision-location\") pod \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") "
Sep 16 18:58:34.590064 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.589843    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tmp-dir\") pod \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\" (UID: \"a14e57a8-fb3b-4abb-86d3-9ae166ed141d\") "
Sep 16 18:58:34.590121 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.590075    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a14e57a8-fb3b-4abb-86d3-9ae166ed141d" (UID: "a14e57a8-fb3b-4abb-86d3-9ae166ed141d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:34.590196 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.590178    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.590539 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.590486    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a14e57a8-fb3b-4abb-86d3-9ae166ed141d" (UID: "a14e57a8-fb3b-4abb-86d3-9ae166ed141d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:34.590828 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.590537    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-home" (OuterVolumeSpecName: "home") pod "a14e57a8-fb3b-4abb-86d3-9ae166ed141d" (UID: "a14e57a8-fb3b-4abb-86d3-9ae166ed141d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:34.592098 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.592073    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kube-api-access-bd6zw" (OuterVolumeSpecName: "kube-api-access-bd6zw") pod "a14e57a8-fb3b-4abb-86d3-9ae166ed141d" (UID: "a14e57a8-fb3b-4abb-86d3-9ae166ed141d"). InnerVolumeSpecName "kube-api-access-bd6zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:34.592189 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.592075    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a14e57a8-fb3b-4abb-86d3-9ae166ed141d" (UID: "a14e57a8-fb3b-4abb-86d3-9ae166ed141d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:34.592189 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.592172    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-dshm" (OuterVolumeSpecName: "dshm") pod "a14e57a8-fb3b-4abb-86d3-9ae166ed141d" (UID: "a14e57a8-fb3b-4abb-86d3-9ae166ed141d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:34.610665 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.610614    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a14e57a8-fb3b-4abb-86d3-9ae166ed141d" (UID: "a14e57a8-fb3b-4abb-86d3-9ae166ed141d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:34.618235 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.618207    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a14e57a8-fb3b-4abb-86d3-9ae166ed141d" (UID: "a14e57a8-fb3b-4abb-86d3-9ae166ed141d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:34.690784 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.690741    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.690784 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.690776    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.690784 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.690791    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.691046 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.690803    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.691046 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.690815    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.691046 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.690825    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd6zw\" (UniqueName: \"kubernetes.io/projected/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-kube-api-access-bd6zw\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.691046 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:34.690835    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14e57a8-fb3b-4abb-86d3-9ae166ed141d-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:35.025819 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.025779    2590 generic.go:356] "Generic (PLEG): container finished" podID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerID="bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415" exitCode=0
Sep 16 18:58:35.026246 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.025842    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"
Sep 16 18:58:35.026246 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.025859    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" event={"ID":"a14e57a8-fb3b-4abb-86d3-9ae166ed141d","Type":"ContainerDied","Data":"bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415"}
Sep 16 18:58:35.026246 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.025923    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7" event={"ID":"a14e57a8-fb3b-4abb-86d3-9ae166ed141d","Type":"ContainerDied","Data":"e91ccbf30c33ee49d6b776b167f1755ecc61bcc589f77a8f604b5d422f9241ad"}
Sep 16 18:58:35.026246 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.025947    2590 scope.go:117] "RemoveContainer" containerID="bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415"
Sep 16 18:58:35.036012 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.035985    2590 scope.go:117] "RemoveContainer" containerID="bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2"
Sep 16 18:58:35.043766 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.043746    2590 scope.go:117] "RemoveContainer" containerID="bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415"
Sep 16 18:58:35.044061 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:58:35.044043    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415\": container with ID starting with bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415 not found: ID does not exist" containerID="bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415"
Sep 16 18:58:35.044118 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.044071    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415"} err="failed to get container status \"bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415\": rpc error: code = NotFound desc = could not find container \"bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415\": container with ID starting with bd2a49d4ea9e85df14695e0b4b697bf8f0f0b2312de8e8f5e08765484a2e8415 not found: ID does not exist"
Sep 16 18:58:35.044118 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.044091    2590 scope.go:117] "RemoveContainer" containerID="bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2"
Sep 16 18:58:35.044370 ip-10-0-143-44 kubenswrapper[2590]: E0916 18:58:35.044354    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2\": container with ID starting with bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2 not found: ID does not exist" containerID="bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2"
Sep 16 18:58:35.044406 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.044378    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2"} err="failed to get container status \"bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2\": rpc error: code = NotFound desc = could not find container \"bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2\": container with ID starting with bbcff6642f1e62da9fea0c1657d0d290c8d81b6a0986284aae13e81b463ce9a2 not found: ID does not exist"
Sep 16 18:58:35.072841 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.072809    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"]
Sep 16 18:58:35.096327 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:35.096292    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7496c4dff4-j6pk7"]
Sep 16 18:58:36.223750 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:36.223717    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.345861 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:36.345816    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.447844 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:36.447804    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:37.025287 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:58:37.025256    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" path="/var/lib/kubelet/pods/a14e57a8-fb3b-4abb-86d3-9ae166ed141d/volumes"
Sep 16 18:59:08.688191 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:08.688110    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:08.784066 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:08.784030    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:12.846529 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.846484    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"]
Sep 16 18:59:12.847006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.846952    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="storage-initializer"
Sep 16 18:59:12.847006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.846970    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="storage-initializer"
Sep 16 18:59:12.847006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.846984    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main"
Sep 16 18:59:12.847006 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.846991    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main"
Sep 16 18:59:12.847219 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.847019    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main"
Sep 16 18:59:12.847219 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.847027    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main"
Sep 16 18:59:12.847219 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.847036    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="storage-initializer"
Sep 16 18:59:12.847219 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.847045    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="storage-initializer"
Sep 16 18:59:12.847219 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.847139    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="b62f8381-e460-49b5-8660-a2301ea31398" containerName="main"
Sep 16 18:59:12.847219 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.847158    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="a14e57a8-fb3b-4abb-86d3-9ae166ed141d" containerName="main"
Sep 16 18:59:12.850069 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.850046    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:12.853371 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.853344    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:59:12.853508 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.853349    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 18:59:12.853766 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.853729    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 18:59:12.853766 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.853732    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 18:59:12.854232 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.854211    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 18:59:12.854770 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.854748    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-6d5ln\""
Sep 16 18:59:12.866489 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.866447    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"]
Sep 16 18:59:12.895158 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.895127    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:12.908280 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.908248    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-dshm\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:12.908371 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.908320    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:12.908371 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.908346    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-tmp-dir\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:12.908616 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.908393    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64df617e-c00a-461d-aa15-915b08342830-tls-certs\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:12.908616 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.908428    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbcg\" (UniqueName: \"kubernetes.io/projected/64df617e-c00a-461d-aa15-915b08342830-kube-api-access-kxbcg\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:12.908616 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.908447    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64df617e-c00a-461d-aa15-915b08342830-cabundle-cert\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:12.908616 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.908465    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-model-cache\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:12.908616 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:12.908481    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-home\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.009191 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009148    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64df617e-c00a-461d-aa15-915b08342830-tls-certs\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.009376 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009214    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbcg\" (UniqueName: \"kubernetes.io/projected/64df617e-c00a-461d-aa15-915b08342830-kube-api-access-kxbcg\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.009376 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009245    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64df617e-c00a-461d-aa15-915b08342830-cabundle-cert\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.009376 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009274    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-model-cache\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.009376 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009299    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-home\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.009376 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009346    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-dshm\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.009611 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009383    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.009611 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009409    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-tmp-dir\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.009850 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009822    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-home\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.010008 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009860    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-tmp-dir\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.010008 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.009956    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-model-cache\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.010115 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.010002    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.010199 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.010181    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64df617e-c00a-461d-aa15-915b08342830-cabundle-cert\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.011823 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.011792    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-dshm\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.011945 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.011909    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64df617e-c00a-461d-aa15-915b08342830-tls-certs\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.027793 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.027769    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbcg\" (UniqueName: \"kubernetes.io/projected/64df617e-c00a-461d-aa15-915b08342830-kube-api-access-kxbcg\") pod \"auth-invalid-token-test-kserve-94545f8c-56mx6\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.162437 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.162347    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:13.255085 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.255049    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:13.453282 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:13.453198    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"]
Sep 16 18:59:13.456438 ip-10-0-143-44 kubenswrapper[2590]: W0916 18:59:13.456413    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64df617e_c00a_461d_aa15_915b08342830.slice/crio-431f346cbdc539d30fb321cd2b156043204afc69c42cca3f8987c059b3560d94 WatchSource:0}: Error finding container 431f346cbdc539d30fb321cd2b156043204afc69c42cca3f8987c059b3560d94: Status 404 returned error can't find the container with id 431f346cbdc539d30fb321cd2b156043204afc69c42cca3f8987c059b3560d94
Sep 16 18:59:14.195595 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:14.195556    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" event={"ID":"64df617e-c00a-461d-aa15-915b08342830","Type":"ContainerStarted","Data":"4b9957e39b01986519dce968b26dd11d0385108d56f3322a63da879169c2bb49"}
Sep 16 18:59:14.196014 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:14.195626    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" event={"ID":"64df617e-c00a-461d-aa15-915b08342830","Type":"ContainerStarted","Data":"431f346cbdc539d30fb321cd2b156043204afc69c42cca3f8987c059b3560d94"}
Sep 16 18:59:16.205398 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:16.205292    2590 generic.go:356] "Generic (PLEG): container finished" podID="64df617e-c00a-461d-aa15-915b08342830" containerID="4b9957e39b01986519dce968b26dd11d0385108d56f3322a63da879169c2bb49" exitCode=0
Sep 16 18:59:16.205398 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:16.205366    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" event={"ID":"64df617e-c00a-461d-aa15-915b08342830","Type":"ContainerDied","Data":"4b9957e39b01986519dce968b26dd11d0385108d56f3322a63da879169c2bb49"}
Sep 16 18:59:17.189646 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:17.189605    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:17.212731 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:17.212703    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" event={"ID":"64df617e-c00a-461d-aa15-915b08342830","Type":"ContainerStarted","Data":"42db15695634ecea4c56b04ecc1eacebe28b60257e1851ea2e6359299afd62c7"}
Sep 16 18:59:17.250301 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:17.250252    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podStartSLOduration=2.967592862 podStartE2EDuration="5.250234201s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.282641339s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:17.248443055 +0000 UTC m=+1172.772901902" watchObservedRunningTime="2026-09-16 18:59:17.250234201 +0000 UTC m=+1172.774693035"
Sep 16 18:59:18.162716 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:18.162679    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:23.162948 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:23.162905    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 18:59:23.169933 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:23.169869    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 18:59:24.219285 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:24.219227    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:24.316940 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:24.316900    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:25.784536 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:25.784494    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:25.870863 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:25.870816    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:26.122643 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:26.122569    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:33.163661 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:33.163613    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 18:59:33.852403 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:33.852344    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:34.156130 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:34.156015    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:34.477866 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:34.477748    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:38.957584 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:38.957497    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:39.090732 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:39.090690    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:40.037645 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:40.037602    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.359793 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:41.359762    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.494085 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:41.494051    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:43.163346 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:43.163300    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 18:59:53.163612 ip-10-0-143-44 kubenswrapper[2590]: I0916 18:59:53.163558    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 19:00:03.163511 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:03.163461    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 19:00:08.465899 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:08.465835    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:08.565100 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:08.565060    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:13.163101 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:13.163055    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 19:00:23.163508 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:23.163461    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 19:00:33.163309 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:33.163254    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 19:00:43.162939 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:43.162898    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 19:00:53.163094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:53.163051    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 19:00:57.514138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:57.514104    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:57.659543 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:00:57.659508    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:01.331249 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:01.331207    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:03.163815 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:03.163765    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 16 19:01:03.978125 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:03.978095    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:06.517274 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:06.517234    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:12.357482 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:12.357450    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:12.432362 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:12.432325    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:13.172817 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:13.172785    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 19:01:13.180701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:13.180674    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 19:01:13.849053 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:13.849022    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:18.222354 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:18.222321    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:22.249131 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:22.249094    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:22.256149 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:22.256118    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"]
Sep 16 19:01:22.327240 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:22.327194    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:35.642431 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:35.642398    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:35.738262 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:35.738229    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:37.278421 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.278376    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main" containerID="cri-o://42db15695634ecea4c56b04ecc1eacebe28b60257e1851ea2e6359299afd62c7" gracePeriod=45
Sep 16 19:01:37.842156 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.842124    2590 generic.go:356] "Generic (PLEG): container finished" podID="64df617e-c00a-461d-aa15-915b08342830" containerID="42db15695634ecea4c56b04ecc1eacebe28b60257e1851ea2e6359299afd62c7" exitCode=0
Sep 16 19:01:37.842307 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.842172    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" event={"ID":"64df617e-c00a-461d-aa15-915b08342830","Type":"ContainerDied","Data":"42db15695634ecea4c56b04ecc1eacebe28b60257e1851ea2e6359299afd62c7"}
Sep 16 19:01:37.932804 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.932780    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 19:01:37.957844 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.957812    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-model-cache\") pod \"64df617e-c00a-461d-aa15-915b08342830\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") "
Sep 16 19:01:37.958037 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.957851    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-kserve-provision-location\") pod \"64df617e-c00a-461d-aa15-915b08342830\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") "
Sep 16 19:01:37.958037 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.957872    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64df617e-c00a-461d-aa15-915b08342830-tls-certs\") pod \"64df617e-c00a-461d-aa15-915b08342830\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") "
Sep 16 19:01:37.958037 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.957905    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64df617e-c00a-461d-aa15-915b08342830-cabundle-cert\") pod \"64df617e-c00a-461d-aa15-915b08342830\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") "
Sep 16 19:01:37.958037 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.957922    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-home\") pod \"64df617e-c00a-461d-aa15-915b08342830\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") "
Sep 16 19:01:37.958269 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.958066    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-dshm\") pod \"64df617e-c00a-461d-aa15-915b08342830\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") "
Sep 16 19:01:37.958269 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.958121    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-tmp-dir\") pod \"64df617e-c00a-461d-aa15-915b08342830\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") "
Sep 16 19:01:37.958269 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.958149    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-model-cache" (OuterVolumeSpecName: "model-cache") pod "64df617e-c00a-461d-aa15-915b08342830" (UID: "64df617e-c00a-461d-aa15-915b08342830"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:37.958269 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.958158    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxbcg\" (UniqueName: \"kubernetes.io/projected/64df617e-c00a-461d-aa15-915b08342830-kube-api-access-kxbcg\") pod \"64df617e-c00a-461d-aa15-915b08342830\" (UID: \"64df617e-c00a-461d-aa15-915b08342830\") "
Sep 16 19:01:37.958492 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.958269    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64df617e-c00a-461d-aa15-915b08342830-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64df617e-c00a-461d-aa15-915b08342830" (UID: "64df617e-c00a-461d-aa15-915b08342830"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:37.958492 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.958483    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:37.958596 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.958503    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64df617e-c00a-461d-aa15-915b08342830-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:37.958652 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.958624    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-home" (OuterVolumeSpecName: "home") pod "64df617e-c00a-461d-aa15-915b08342830" (UID: "64df617e-c00a-461d-aa15-915b08342830"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:37.960554 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.960519    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-dshm" (OuterVolumeSpecName: "dshm") pod "64df617e-c00a-461d-aa15-915b08342830" (UID: "64df617e-c00a-461d-aa15-915b08342830"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:37.960731 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.960575    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64df617e-c00a-461d-aa15-915b08342830-kube-api-access-kxbcg" (OuterVolumeSpecName: "kube-api-access-kxbcg") pod "64df617e-c00a-461d-aa15-915b08342830" (UID: "64df617e-c00a-461d-aa15-915b08342830"). InnerVolumeSpecName "kube-api-access-kxbcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:37.960731 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.960577    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64df617e-c00a-461d-aa15-915b08342830-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64df617e-c00a-461d-aa15-915b08342830" (UID: "64df617e-c00a-461d-aa15-915b08342830"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:37.979164 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.979089    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64df617e-c00a-461d-aa15-915b08342830" (UID: "64df617e-c00a-461d-aa15-915b08342830"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:37.987170 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:37.987135    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64df617e-c00a-461d-aa15-915b08342830" (UID: "64df617e-c00a-461d-aa15-915b08342830"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:38.059189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.059098    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.059189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.059136    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64df617e-c00a-461d-aa15-915b08342830-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.059189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.059148    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.059189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.059156    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.059189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.059164    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64df617e-c00a-461d-aa15-915b08342830-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.059189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.059172    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxbcg\" (UniqueName: \"kubernetes.io/projected/64df617e-c00a-461d-aa15-915b08342830-kube-api-access-kxbcg\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.847891 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.847856    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6" event={"ID":"64df617e-c00a-461d-aa15-915b08342830","Type":"ContainerDied","Data":"431f346cbdc539d30fb321cd2b156043204afc69c42cca3f8987c059b3560d94"}
Sep 16 19:01:38.848295 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.847932    2590 scope.go:117] "RemoveContainer" containerID="42db15695634ecea4c56b04ecc1eacebe28b60257e1851ea2e6359299afd62c7"
Sep 16 19:01:38.848295 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.847863    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"
Sep 16 19:01:38.856908 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.856871    2590 scope.go:117] "RemoveContainer" containerID="4b9957e39b01986519dce968b26dd11d0385108d56f3322a63da879169c2bb49"
Sep 16 19:01:38.881165 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.878549    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"]
Sep 16 19:01:38.883273 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:38.883249    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-94545f8c-56mx6"]
Sep 16 19:01:39.025154 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:01:39.025121    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64df617e-c00a-461d-aa15-915b08342830" path="/var/lib/kubelet/pods/64df617e-c00a-461d-aa15-915b08342830/volumes"
Sep 16 19:02:11.939914 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:11.939768    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:12.058017 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:12.057969    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:16.462851 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.462815    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"]
Sep 16 19:02:16.463264 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.463101    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="storage-initializer"
Sep 16 19:02:16.463264 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.463112    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="storage-initializer"
Sep 16 19:02:16.463264 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.463131    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main"
Sep 16 19:02:16.463264 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.463136    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main"
Sep 16 19:02:16.463264 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.463182    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="64df617e-c00a-461d-aa15-915b08342830" containerName="main"
Sep 16 19:02:16.464992 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.464973    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.467692 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.467659    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:02:16.467820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.467672    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 19:02:16.468048 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.468031    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:02:16.468048 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.468043    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:02:16.468332 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.468314    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-5ccsq\""
Sep 16 19:02:16.468418 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.468353    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:02:16.483594 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.483568    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"]
Sep 16 19:02:16.491198 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.491172    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:16.571672 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.571630    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-model-cache\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.571860 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.571689    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-home\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.571860 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.571751    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wp2\" (UniqueName: \"kubernetes.io/projected/97291677-7931-4217-9358-f7011ca5b5b6-kube-api-access-99wp2\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.571860 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.571786    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-tmp-dir\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.571860 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.571807    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97291677-7931-4217-9358-f7011ca5b5b6-cabundle-cert\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.571860 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.571840    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-dshm\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.572139 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.571905    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97291677-7931-4217-9358-f7011ca5b5b6-tls-certs\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.572139 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.571979    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-kserve-provision-location\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.672740 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.672703    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-model-cache\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.672740 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.672746    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-home\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.672764    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wp2\" (UniqueName: \"kubernetes.io/projected/97291677-7931-4217-9358-f7011ca5b5b6-kube-api-access-99wp2\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.672783    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-tmp-dir\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.672800    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97291677-7931-4217-9358-f7011ca5b5b6-cabundle-cert\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.672839    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-dshm\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.672853    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97291677-7931-4217-9358-f7011ca5b5b6-tls-certs\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.672873    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-kserve-provision-location\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673307 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.673180    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-model-cache\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673307 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.673253    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-tmp-dir\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673307 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.673276    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-home\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673457 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.673340    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-kserve-provision-location\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.673560 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.673540    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97291677-7931-4217-9358-f7011ca5b5b6-cabundle-cert\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.675435 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.675409    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-dshm\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.675554 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.675534    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97291677-7931-4217-9358-f7011ca5b5b6-tls-certs\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.681270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.681247    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wp2\" (UniqueName: \"kubernetes.io/projected/97291677-7931-4217-9358-f7011ca5b5b6-kube-api-access-99wp2\") pod \"auth-disabled-test-kserve-c794d68d8-9v4k7\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.775724 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.775692    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:16.787491 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:16.787463    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:17.060507 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:17.060484    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"]
Sep 16 19:02:17.061727 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:02:17.061699    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97291677_7931_4217_9358_f7011ca5b5b6.slice/crio-bc69357c0c30f2049c7fb46f7dfef4cf8fbb0da2a4e311db457f74e224bab048 WatchSource:0}: Error finding container bc69357c0c30f2049c7fb46f7dfef4cf8fbb0da2a4e311db457f74e224bab048: Status 404 returned error can't find the container with id bc69357c0c30f2049c7fb46f7dfef4cf8fbb0da2a4e311db457f74e224bab048
Sep 16 19:02:18.018467 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:18.018425    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" event={"ID":"97291677-7931-4217-9358-f7011ca5b5b6","Type":"ContainerStarted","Data":"b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d"}
Sep 16 19:02:18.019094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:18.018479    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" event={"ID":"97291677-7931-4217-9358-f7011ca5b5b6","Type":"ContainerStarted","Data":"bc69357c0c30f2049c7fb46f7dfef4cf8fbb0da2a4e311db457f74e224bab048"}
Sep 16 19:02:19.795068 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:19.795033    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:20.031175 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:20.031140    2590 generic.go:356] "Generic (PLEG): container finished" podID="97291677-7931-4217-9358-f7011ca5b5b6" containerID="b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d" exitCode=0
Sep 16 19:02:20.031340 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:20.031216    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" event={"ID":"97291677-7931-4217-9358-f7011ca5b5b6","Type":"ContainerDied","Data":"b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d"}
Sep 16 19:02:21.037718 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:21.037686    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" event={"ID":"97291677-7931-4217-9358-f7011ca5b5b6","Type":"ContainerStarted","Data":"d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8"}
Sep 16 19:02:21.064855 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:21.064792    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podStartSLOduration=2.792598261 podStartE2EDuration="5.064769711s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.27217145s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:21.06216858 +0000 UTC m=+1356.586627413" watchObservedRunningTime="2026-09-16 19:02:21.064769711 +0000 UTC m=+1356.589228546"
Sep 16 19:02:21.776857 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:21.776811    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:25.583854 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:25.583820    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:25.680385 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:25.680349    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:25.840225 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:25.840141    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:26.433547 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:26.433517    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:26.536500 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:26.536464    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:26.776320 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:26.776289    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:02:26.777956 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:26.777931    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:02:29.352286 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:29.352252    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:32.418596 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:32.418563    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:36.776962 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:36.776922    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:02:38.627117 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:38.627081    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:38.734342 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:38.734310    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:44.612807 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.612776    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:44.831023 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.830987    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"]
Sep 16 19:02:44.834514 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.834489    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:02:44.838523 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.838485    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:02:44.838523 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.838485    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-c2qsv\""
Sep 16 19:02:44.838777 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.838487    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:02:44.838777 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.838541    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-2dkzp\""
Sep 16 19:02:44.838777 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.838485    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:02:44.847108 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.847085    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"]
Sep 16 19:02:44.851789 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.851767    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:44.906222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.906131    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18c3823-b3a2-4b8a-9f3f-02748980aac2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs\" (UID: \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:02:44.906222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:44.906173    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcj7\" (UniqueName: \"kubernetes.io/projected/a18c3823-b3a2-4b8a-9f3f-02748980aac2-kube-api-access-glcj7\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs\" (UID: \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:02:45.006537 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.006503    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18c3823-b3a2-4b8a-9f3f-02748980aac2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs\" (UID: \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:02:45.006702 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.006543    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcj7\" (UniqueName: \"kubernetes.io/projected/a18c3823-b3a2-4b8a-9f3f-02748980aac2-kube-api-access-glcj7\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs\" (UID: \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:02:45.009229 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.009205    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:02:45.015139 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.015123    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:02:45.019252 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.019235    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18c3823-b3a2-4b8a-9f3f-02748980aac2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs\" (UID: \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:02:45.025999 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.025979    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:02:45.035913 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.035857    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcj7\" (UniqueName: \"kubernetes.io/projected/a18c3823-b3a2-4b8a-9f3f-02748980aac2-kube-api-access-glcj7\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs\" (UID: \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:02:45.151014 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.150974    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-c2qsv\""
Sep 16 19:02:45.161770 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.161701    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-2dkzp\""
Sep 16 19:02:45.169752 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.169724    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:02:45.448757 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.448733    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"]
Sep 16 19:02:45.451333 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:02:45.451298    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18c3823_b3a2_4b8a_9f3f_02748980aac2.slice/crio-21c403663cebafbe9b7506fcefc263913e7ae5696648b942296aabd774c31453 WatchSource:0}: Error finding container 21c403663cebafbe9b7506fcefc263913e7ae5696648b942296aabd774c31453: Status 404 returned error can't find the container with id 21c403663cebafbe9b7506fcefc263913e7ae5696648b942296aabd774c31453
Sep 16 19:02:45.453848 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:45.453826    2590 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:02:46.149931 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:46.149872    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" event={"ID":"a18c3823-b3a2-4b8a-9f3f-02748980aac2","Type":"ContainerStarted","Data":"21c403663cebafbe9b7506fcefc263913e7ae5696648b942296aabd774c31453"}
Sep 16 19:02:46.776511 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:46.776470    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:02:49.165936 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:49.165871    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" event={"ID":"a18c3823-b3a2-4b8a-9f3f-02748980aac2","Type":"ContainerStarted","Data":"89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da"}
Sep 16 19:02:49.166307 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:49.165965    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:02:49.186443 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:49.186391    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" podStartSLOduration=2.44152652 podStartE2EDuration="5.186373054s" totalImagesPullingTime="2.744846534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:49.18544353 +0000 UTC m=+1384.709902368" watchObservedRunningTime="2026-09-16 19:02:49.186373054 +0000 UTC m=+1384.710831888"
Sep 16 19:02:56.776186 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:02:56.776143    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:03:01.002737 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:01.002700    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:01.127583 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:01.127549    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:02.248498 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:02.248461    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:06.776472 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:06.776426    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:03:16.777122 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:16.777075    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:03:18.651963 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:18.651926    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"]
Sep 16 19:03:18.653703 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:18.653671    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:18.744168 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:18.744134    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:18.844372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:18.844342    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:26.777024 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:26.776979    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:03:28.654247 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:28.654209    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:33.674130 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:33.674073    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerName="main" containerID="cri-o://89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da" gracePeriod=45
Sep 16 19:03:36.776972 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:36.776864    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:03:38.654694 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:38.654639    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:46.776409 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:46.776345    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:03:48.654003 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:48.653969    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:56.776358 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:56.776318    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:03:57.199282 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:57.199163    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:58.654152 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:03:58.654113    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:03.922604 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:03.922567    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:04:04.008672 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.008572    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glcj7\" (UniqueName: \"kubernetes.io/projected/a18c3823-b3a2-4b8a-9f3f-02748980aac2-kube-api-access-glcj7\") pod \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\" (UID: \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\") "
Sep 16 19:04:04.008672 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.008650    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18c3823-b3a2-4b8a-9f3f-02748980aac2-tls-certs\") pod \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\" (UID: \"a18c3823-b3a2-4b8a-9f3f-02748980aac2\") "
Sep 16 19:04:04.011090 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.011059    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18c3823-b3a2-4b8a-9f3f-02748980aac2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a18c3823-b3a2-4b8a-9f3f-02748980aac2" (UID: "a18c3823-b3a2-4b8a-9f3f-02748980aac2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:04.011210 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.011059    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18c3823-b3a2-4b8a-9f3f-02748980aac2-kube-api-access-glcj7" (OuterVolumeSpecName: "kube-api-access-glcj7") pod "a18c3823-b3a2-4b8a-9f3f-02748980aac2" (UID: "a18c3823-b3a2-4b8a-9f3f-02748980aac2"). InnerVolumeSpecName "kube-api-access-glcj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:04.110343 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.110281    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18c3823-b3a2-4b8a-9f3f-02748980aac2-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:04.110343 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.110345    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glcj7\" (UniqueName: \"kubernetes.io/projected/a18c3823-b3a2-4b8a-9f3f-02748980aac2-kube-api-access-glcj7\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:04.512561 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.512524    2590 generic.go:356] "Generic (PLEG): container finished" podID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerID="89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da" exitCode=0
Sep 16 19:04:04.512561 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.512559    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"
Sep 16 19:04:04.512807 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.512645    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" event={"ID":"a18c3823-b3a2-4b8a-9f3f-02748980aac2","Type":"ContainerDied","Data":"89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da"}
Sep 16 19:04:04.512807 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.512710    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs" event={"ID":"a18c3823-b3a2-4b8a-9f3f-02748980aac2","Type":"ContainerDied","Data":"21c403663cebafbe9b7506fcefc263913e7ae5696648b942296aabd774c31453"}
Sep 16 19:04:04.512807 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.512741    2590 scope.go:117] "RemoveContainer" containerID="89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da"
Sep 16 19:04:04.522189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.522163    2590 scope.go:117] "RemoveContainer" containerID="89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da"
Sep 16 19:04:04.522501 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:04:04.522481    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da\": container with ID starting with 89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da not found: ID does not exist" containerID="89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da"
Sep 16 19:04:04.522556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.522513    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da"} err="failed to get container status \"89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da\": rpc error: code = NotFound desc = could not find container \"89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da\": container with ID starting with 89738d7b91efa33eaec22fdf5ec5047acdabdfa6bdc11815d239c5d89edd42da not found: ID does not exist"
Sep 16 19:04:04.543508 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.543464    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"]
Sep 16 19:04:04.545793 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:04.545760    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5xcrfs"]
Sep 16 19:04:05.026396 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:05.026356    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" path="/var/lib/kubelet/pods/a18c3823-b3a2-4b8a-9f3f-02748980aac2/volumes"
Sep 16 19:04:06.776236 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:06.776191    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:04:08.286631 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:08.286598    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:08.439203 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:08.439169    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:14.883530 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:14.883494    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:16.787024 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:16.786995    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:04:16.794742 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:16.794723    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:04:20.577414 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.577379    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:20.746313 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.746285    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"]
Sep 16 19:04:20.746585 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.746573    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerName="main"
Sep 16 19:04:20.746635 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.746588    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerName="main"
Sep 16 19:04:20.746670 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.746656    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18c3823-b3a2-4b8a-9f3f-02748980aac2" containerName="main"
Sep 16 19:04:20.749322 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.749304    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.753240 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.753208    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 19:04:20.753240 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.753224    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 19:04:20.753428 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.753255    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 19:04:20.753428 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.753309    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-hh7hx\""
Sep 16 19:04:20.753536 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.753442    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-55n52\""
Sep 16 19:04:20.763064 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.763040    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"]
Sep 16 19:04:20.773860 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.773832    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:20.848978 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.848860    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/735ee18c-ad9d-420e-9f01-7df9425c27b5-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.849155 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.849016    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.849155 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.849052    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.849155 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.849083    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdz6z\" (UniqueName: \"kubernetes.io/projected/735ee18c-ad9d-420e-9f01-7df9425c27b5-kube-api-access-kdz6z\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.950247 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.950206    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.950247 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.950248    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.950755 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.950266    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdz6z\" (UniqueName: \"kubernetes.io/projected/735ee18c-ad9d-420e-9f01-7df9425c27b5-kube-api-access-kdz6z\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.950755 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.950313    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/735ee18c-ad9d-420e-9f01-7df9425c27b5-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.950755 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.950681    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.950755 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.950708    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.952996 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.952963    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/735ee18c-ad9d-420e-9f01-7df9425c27b5-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:20.963840 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:20.963810    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdz6z\" (UniqueName: \"kubernetes.io/projected/735ee18c-ad9d-420e-9f01-7df9425c27b5-kube-api-access-kdz6z\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:21.005789 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:21.005663    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:21.059333 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:21.059302    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:21.071714 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:21.071685    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:21.340048 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:21.340013    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"]
Sep 16 19:04:21.343154 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:04:21.343113    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735ee18c_ad9d_420e_9f01_7df9425c27b5.slice/crio-7293f126a033bee37fa2ed68fd179fd816d1b8114c0315a46c6253820d5cf098 WatchSource:0}: Error finding container 7293f126a033bee37fa2ed68fd179fd816d1b8114c0315a46c6253820d5cf098: Status 404 returned error can't find the container with id 7293f126a033bee37fa2ed68fd179fd816d1b8114c0315a46c6253820d5cf098
Sep 16 19:04:21.596158 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:21.596121    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" event={"ID":"735ee18c-ad9d-420e-9f01-7df9425c27b5","Type":"ContainerStarted","Data":"2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03"}
Sep 16 19:04:21.596563 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:21.596214    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" event={"ID":"735ee18c-ad9d-420e-9f01-7df9425c27b5","Type":"ContainerStarted","Data":"7293f126a033bee37fa2ed68fd179fd816d1b8114c0315a46c6253820d5cf098"}
Sep 16 19:04:21.596563 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:21.596262    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:21.616477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:21.616361    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" podStartSLOduration=1.616342297 podStartE2EDuration="1.616342297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:21.614717316 +0000 UTC m=+1477.139176151" watchObservedRunningTime="2026-09-16 19:04:21.616342297 +0000 UTC m=+1477.140801130"
Sep 16 19:04:28.586849 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:28.586812    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:29.471464 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:29.471430    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:31.110346 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:31.110312    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:31.111257 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:31.111234    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"]
Sep 16 19:04:31.183478 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:31.183422    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:31.372677 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:31.372545    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:42.614376 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:42.614339    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:46.134867 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.134804    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" containerID="cri-o://d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8" gracePeriod=45
Sep 16 19:04:46.353637 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.353594    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:46.420016 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.419929    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:46.788493 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.788448    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 19:04:46.882430 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.882405    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:04:46.976305 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976227    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-model-cache\") pod \"97291677-7931-4217-9358-f7011ca5b5b6\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") "
Sep 16 19:04:46.976305 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976294    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99wp2\" (UniqueName: \"kubernetes.io/projected/97291677-7931-4217-9358-f7011ca5b5b6-kube-api-access-99wp2\") pod \"97291677-7931-4217-9358-f7011ca5b5b6\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") "
Sep 16 19:04:46.976524 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976320    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-dshm\") pod \"97291677-7931-4217-9358-f7011ca5b5b6\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") "
Sep 16 19:04:46.976524 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976378    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-home\") pod \"97291677-7931-4217-9358-f7011ca5b5b6\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") "
Sep 16 19:04:46.976524 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976405    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-kserve-provision-location\") pod \"97291677-7931-4217-9358-f7011ca5b5b6\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") "
Sep 16 19:04:46.976524 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976433    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97291677-7931-4217-9358-f7011ca5b5b6-cabundle-cert\") pod \"97291677-7931-4217-9358-f7011ca5b5b6\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") "
Sep 16 19:04:46.976524 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976467    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-tmp-dir\") pod \"97291677-7931-4217-9358-f7011ca5b5b6\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") "
Sep 16 19:04:46.976524 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976509    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97291677-7931-4217-9358-f7011ca5b5b6-tls-certs\") pod \"97291677-7931-4217-9358-f7011ca5b5b6\" (UID: \"97291677-7931-4217-9358-f7011ca5b5b6\") "
Sep 16 19:04:46.976822 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976530    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "97291677-7931-4217-9358-f7011ca5b5b6" (UID: "97291677-7931-4217-9358-f7011ca5b5b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:46.976822 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.976748    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:46.977354 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.977276    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-home" (OuterVolumeSpecName: "home") pod "97291677-7931-4217-9358-f7011ca5b5b6" (UID: "97291677-7931-4217-9358-f7011ca5b5b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:46.977488 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.977393    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97291677-7931-4217-9358-f7011ca5b5b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "97291677-7931-4217-9358-f7011ca5b5b6" (UID: "97291677-7931-4217-9358-f7011ca5b5b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:46.979211 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.979186    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97291677-7931-4217-9358-f7011ca5b5b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97291677-7931-4217-9358-f7011ca5b5b6" (UID: "97291677-7931-4217-9358-f7011ca5b5b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:46.979326 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.979263    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97291677-7931-4217-9358-f7011ca5b5b6-kube-api-access-99wp2" (OuterVolumeSpecName: "kube-api-access-99wp2") pod "97291677-7931-4217-9358-f7011ca5b5b6" (UID: "97291677-7931-4217-9358-f7011ca5b5b6"). InnerVolumeSpecName "kube-api-access-99wp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:46.979388 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.979321    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-dshm" (OuterVolumeSpecName: "dshm") pod "97291677-7931-4217-9358-f7011ca5b5b6" (UID: "97291677-7931-4217-9358-f7011ca5b5b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:46.989968 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:46.989873    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97291677-7931-4217-9358-f7011ca5b5b6" (UID: "97291677-7931-4217-9358-f7011ca5b5b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:47.000783 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.000754    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97291677-7931-4217-9358-f7011ca5b5b6" (UID: "97291677-7931-4217-9358-f7011ca5b5b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:47.077532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.077502    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wp2\" (UniqueName: \"kubernetes.io/projected/97291677-7931-4217-9358-f7011ca5b5b6-kube-api-access-99wp2\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:47.077532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.077529    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:47.077532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.077540    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:47.077770 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.077549    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:47.077770 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.077558    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97291677-7931-4217-9358-f7011ca5b5b6-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:47.077770 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.077585    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97291677-7931-4217-9358-f7011ca5b5b6-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:47.077770 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.077593    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97291677-7931-4217-9358-f7011ca5b5b6-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:47.709785 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.709745    2590 generic.go:356] "Generic (PLEG): container finished" podID="97291677-7931-4217-9358-f7011ca5b5b6" containerID="d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8" exitCode=0
Sep 16 19:04:47.710415 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.709803    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"
Sep 16 19:04:47.710415 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.709804    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" event={"ID":"97291677-7931-4217-9358-f7011ca5b5b6","Type":"ContainerDied","Data":"d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8"}
Sep 16 19:04:47.710415 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.709857    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7" event={"ID":"97291677-7931-4217-9358-f7011ca5b5b6","Type":"ContainerDied","Data":"bc69357c0c30f2049c7fb46f7dfef4cf8fbb0da2a4e311db457f74e224bab048"}
Sep 16 19:04:47.710415 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.709911    2590 scope.go:117] "RemoveContainer" containerID="d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8"
Sep 16 19:04:47.718339 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.718321    2590 scope.go:117] "RemoveContainer" containerID="b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d"
Sep 16 19:04:47.725820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.725800    2590 scope.go:117] "RemoveContainer" containerID="d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8"
Sep 16 19:04:47.726124 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:04:47.726105    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8\": container with ID starting with d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8 not found: ID does not exist" containerID="d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8"
Sep 16 19:04:47.726182 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.726136    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8"} err="failed to get container status \"d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8\": rpc error: code = NotFound desc = could not find container \"d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8\": container with ID starting with d5af7b6df7d069c23fa347fb99a70665a0dac8d70307b39638ea90f0ce5373d8 not found: ID does not exist"
Sep 16 19:04:47.726182 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.726154    2590 scope.go:117] "RemoveContainer" containerID="b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d"
Sep 16 19:04:47.726427 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:04:47.726408    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d\": container with ID starting with b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d not found: ID does not exist" containerID="b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d"
Sep 16 19:04:47.726468 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.726434    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d"} err="failed to get container status \"b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d\": rpc error: code = NotFound desc = could not find container \"b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d\": container with ID starting with b6a2bda65ee977349d3f6c05b63f9bd7a8a0909849a1e501b4152994788b4f8d not found: ID does not exist"
Sep 16 19:04:47.743812 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.743792    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"]
Sep 16 19:04:47.760920 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:47.760895    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c794d68d8-9v4k7"]
Sep 16 19:04:49.026218 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:49.026187    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97291677-7931-4217-9358-f7011ca5b5b6" path="/var/lib/kubelet/pods/97291677-7931-4217-9358-f7011ca5b5b6/volumes"
Sep 16 19:04:52.602489 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:52.602462    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:04:54.376033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:54.376001    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"]
Sep 16 19:04:55.015486 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:55.015453    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:55.120955 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:04:55.120919    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:02.602747 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:02.602709    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" podUID="735ee18c-ad9d-420e-9f01-7df9425c27b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:09.395703 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:09.395602    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" podUID="735ee18c-ad9d-420e-9f01-7df9425c27b5" containerName="main" containerID="cri-o://2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03" gracePeriod=45
Sep 16 19:05:12.602933 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:12.602872    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" podUID="735ee18c-ad9d-420e-9f01-7df9425c27b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:18.885971 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:18.885936    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:19.017611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:19.017576    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:19.155841 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:19.155747    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:22.602658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:22.602616    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" podUID="735ee18c-ad9d-420e-9f01-7df9425c27b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:22.603103 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:22.602723    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:05:26.094457 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:26.094419    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:32.602835 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:32.602795    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" podUID="735ee18c-ad9d-420e-9f01-7df9425c27b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:32.694108 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:32.694073    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:32.791297 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:32.791261    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:33.011465 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:33.011381    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:39.641920 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.641869    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:05:39.695248 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.695144    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-tokenizer-cache\") pod \"735ee18c-ad9d-420e-9f01-7df9425c27b5\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") "
Sep 16 19:05:39.695248 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.695196    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdz6z\" (UniqueName: \"kubernetes.io/projected/735ee18c-ad9d-420e-9f01-7df9425c27b5-kube-api-access-kdz6z\") pod \"735ee18c-ad9d-420e-9f01-7df9425c27b5\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") "
Sep 16 19:05:39.695471 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.695276    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-cachi2-cache\") pod \"735ee18c-ad9d-420e-9f01-7df9425c27b5\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") "
Sep 16 19:05:39.695471 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.695312    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/735ee18c-ad9d-420e-9f01-7df9425c27b5-tls-certs\") pod \"735ee18c-ad9d-420e-9f01-7df9425c27b5\" (UID: \"735ee18c-ad9d-420e-9f01-7df9425c27b5\") "
Sep 16 19:05:39.695471 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.695415    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "735ee18c-ad9d-420e-9f01-7df9425c27b5" (UID: "735ee18c-ad9d-420e-9f01-7df9425c27b5"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:39.695587 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.695534    2590 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-tokenizer-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:05:39.695587 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.695531    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "735ee18c-ad9d-420e-9f01-7df9425c27b5" (UID: "735ee18c-ad9d-420e-9f01-7df9425c27b5"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:39.697721 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.697684    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735ee18c-ad9d-420e-9f01-7df9425c27b5-kube-api-access-kdz6z" (OuterVolumeSpecName: "kube-api-access-kdz6z") pod "735ee18c-ad9d-420e-9f01-7df9425c27b5" (UID: "735ee18c-ad9d-420e-9f01-7df9425c27b5"). InnerVolumeSpecName "kube-api-access-kdz6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:39.697721 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.697700    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/735ee18c-ad9d-420e-9f01-7df9425c27b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "735ee18c-ad9d-420e-9f01-7df9425c27b5" (UID: "735ee18c-ad9d-420e-9f01-7df9425c27b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:39.796532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.796475    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/735ee18c-ad9d-420e-9f01-7df9425c27b5-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:05:39.796532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.796524    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdz6z\" (UniqueName: \"kubernetes.io/projected/735ee18c-ad9d-420e-9f01-7df9425c27b5-kube-api-access-kdz6z\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:05:39.796532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.796534    2590 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/735ee18c-ad9d-420e-9f01-7df9425c27b5-cachi2-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:05:39.940941 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.940906    2590 generic.go:356] "Generic (PLEG): container finished" podID="735ee18c-ad9d-420e-9f01-7df9425c27b5" containerID="2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03" exitCode=0
Sep 16 19:05:39.941131 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.940950    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" event={"ID":"735ee18c-ad9d-420e-9f01-7df9425c27b5","Type":"ContainerDied","Data":"2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03"}
Sep 16 19:05:39.941131 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.940978    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"
Sep 16 19:05:39.941131 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.941029    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt" event={"ID":"735ee18c-ad9d-420e-9f01-7df9425c27b5","Type":"ContainerDied","Data":"7293f126a033bee37fa2ed68fd179fd816d1b8114c0315a46c6253820d5cf098"}
Sep 16 19:05:39.941131 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.941051    2590 scope.go:117] "RemoveContainer" containerID="2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03"
Sep 16 19:05:39.950240 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.950223    2590 scope.go:117] "RemoveContainer" containerID="2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03"
Sep 16 19:05:39.950536 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:05:39.950515    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03\": container with ID starting with 2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03 not found: ID does not exist" containerID="2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03"
Sep 16 19:05:39.950594 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.950544    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03"} err="failed to get container status \"2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03\": rpc error: code = NotFound desc = could not find container \"2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03\": container with ID starting with 2c2b7f475bd81344a2ee58802fe6c889a87a894fa6e06b450545fcd08f6a8b03 not found: ID does not exist"
Sep 16 19:05:39.965138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.965101    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"]
Sep 16 19:05:39.968918 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:39.968853    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcbbkbt"]
Sep 16 19:05:41.026378 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:41.026335    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="735ee18c-ad9d-420e-9f01-7df9425c27b5" path="/var/lib/kubelet/pods/735ee18c-ad9d-420e-9f01-7df9425c27b5/volumes"
Sep 16 19:05:44.473324 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:44.473289    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:44.653298 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:44.653269    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:44.762577 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:44.762484    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:50.114423 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:50.114388    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:52.321658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:52.321608    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:58.602448 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:58.602410    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:58.700047 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:05:58.700009    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:16.799275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.799235    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"]
Sep 16 19:06:16.801658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.799523    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="735ee18c-ad9d-420e-9f01-7df9425c27b5" containerName="main"
Sep 16 19:06:16.801658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.799534    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="735ee18c-ad9d-420e-9f01-7df9425c27b5" containerName="main"
Sep 16 19:06:16.801658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.799552    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main"
Sep 16 19:06:16.801658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.799558    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main"
Sep 16 19:06:16.801658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.799566    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="storage-initializer"
Sep 16 19:06:16.801658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.799572    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="storage-initializer"
Sep 16 19:06:16.801658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.799621    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="97291677-7931-4217-9358-f7011ca5b5b6" containerName="main"
Sep 16 19:06:16.801658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.799631    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="735ee18c-ad9d-420e-9f01-7df9425c27b5" containerName="main"
Sep 16 19:06:16.802638 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.802621    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.806645 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.806623    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-dprcq\""
Sep 16 19:06:16.806772 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.806624    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:06:16.806772 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.806730    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:06:16.806959 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.806939    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:06:16.812817 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.812796    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"]
Sep 16 19:06:16.822786 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.822757    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.822913 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.822795    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9239f0c-7f62-494b-9c6f-e105b54c195b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.822913 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.822842    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rprl\" (UniqueName: \"kubernetes.io/projected/d9239f0c-7f62-494b-9c6f-e105b54c195b-kube-api-access-4rprl\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.822998 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.822911    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.822998 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.822940    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.822998 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.822973    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.832553 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.832527    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:16.923709 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.923673    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.923930 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.923712    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.923930 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.923750    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.923930 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.923777    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.923930 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.923802    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9239f0c-7f62-494b-9c6f-e105b54c195b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.923930 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.923840    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rprl\" (UniqueName: \"kubernetes.io/projected/d9239f0c-7f62-494b-9c6f-e105b54c195b-kube-api-access-4rprl\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.924234 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.924209    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.924300 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.924252    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.924300 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.924281    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.926156 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.926134    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.926429 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.926410    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9239f0c-7f62-494b-9c6f-e105b54c195b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:16.933017 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:16.932992    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rprl\" (UniqueName: \"kubernetes.io/projected/d9239f0c-7f62-494b-9c6f-e105b54c195b-kube-api-access-4rprl\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:17.067473 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.067394    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"]
Sep 16 19:06:17.070637 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.070619    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:06:17.074109 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.074088    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-hh24g\""
Sep 16 19:06:17.086226 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.086197    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"]
Sep 16 19:06:17.113735 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.113705    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:17.125052 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.125022    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjgz8\" (UniqueName: \"kubernetes.io/projected/ba609aac-f16c-4f11-aac0-5febc82aaebd-kube-api-access-gjgz8\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx\" (UID: \"ba609aac-f16c-4f11-aac0-5febc82aaebd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:06:17.125146 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.125075    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba609aac-f16c-4f11-aac0-5febc82aaebd-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx\" (UID: \"ba609aac-f16c-4f11-aac0-5febc82aaebd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:06:17.226186 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.226146    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba609aac-f16c-4f11-aac0-5febc82aaebd-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx\" (UID: \"ba609aac-f16c-4f11-aac0-5febc82aaebd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:06:17.226360 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.226231    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgz8\" (UniqueName: \"kubernetes.io/projected/ba609aac-f16c-4f11-aac0-5febc82aaebd-kube-api-access-gjgz8\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx\" (UID: \"ba609aac-f16c-4f11-aac0-5febc82aaebd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:06:17.228752 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.228733    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba609aac-f16c-4f11-aac0-5febc82aaebd-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx\" (UID: \"ba609aac-f16c-4f11-aac0-5febc82aaebd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:06:17.238692 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.238669    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgz8\" (UniqueName: \"kubernetes.io/projected/ba609aac-f16c-4f11-aac0-5febc82aaebd-kube-api-access-gjgz8\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx\" (UID: \"ba609aac-f16c-4f11-aac0-5febc82aaebd\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:06:17.380314 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.380235    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:06:17.391634 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.391525    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"]
Sep 16 19:06:17.394277 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:06:17.394100    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9239f0c_7f62_494b_9c6f_e105b54c195b.slice/crio-e4d6600b91695e6b952484414e719f40a713268b0211ced7efb504eb199c3441 WatchSource:0}: Error finding container e4d6600b91695e6b952484414e719f40a713268b0211ced7efb504eb199c3441: Status 404 returned error can't find the container with id e4d6600b91695e6b952484414e719f40a713268b0211ced7efb504eb199c3441
Sep 16 19:06:17.668184 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:06:17.668154    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba609aac_f16c_4f11_aac0_5febc82aaebd.slice/crio-540e40863ccd0cd8a4bd8c42a240788d6709af54bd505ec93d820c5755a3f149 WatchSource:0}: Error finding container 540e40863ccd0cd8a4bd8c42a240788d6709af54bd505ec93d820c5755a3f149: Status 404 returned error can't find the container with id 540e40863ccd0cd8a4bd8c42a240788d6709af54bd505ec93d820c5755a3f149
Sep 16 19:06:17.668908 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:17.668864    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"]
Sep 16 19:06:18.106521 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:18.106240    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" event={"ID":"ba609aac-f16c-4f11-aac0-5febc82aaebd","Type":"ContainerStarted","Data":"0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2"}
Sep 16 19:06:18.106521 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:18.106302    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" event={"ID":"ba609aac-f16c-4f11-aac0-5febc82aaebd","Type":"ContainerStarted","Data":"540e40863ccd0cd8a4bd8c42a240788d6709af54bd505ec93d820c5755a3f149"}
Sep 16 19:06:18.106521 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:18.106331    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:06:18.107962 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:18.107935    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww" event={"ID":"d9239f0c-7f62-494b-9c6f-e105b54c195b","Type":"ContainerStarted","Data":"e4d6600b91695e6b952484414e719f40a713268b0211ced7efb504eb199c3441"}
Sep 16 19:06:18.148101 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:18.148043    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" podStartSLOduration=1.148023801 podStartE2EDuration="1.148023801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:18.145533719 +0000 UTC m=+1593.669992552" watchObservedRunningTime="2026-09-16 19:06:18.148023801 +0000 UTC m=+1593.672482635"
Sep 16 19:06:18.439684 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:18.439563    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:20.119547 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:20.119507    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww" event={"ID":"d9239f0c-7f62-494b-9c6f-e105b54c195b","Type":"ContainerStarted","Data":"591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112"}
Sep 16 19:06:20.140058 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:20.140002    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww" podStartSLOduration=2.470325895 podStartE2EDuration="4.139984195s" totalImagesPullingTime="1.6696583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:20.137978869 +0000 UTC m=+1595.662437705" watchObservedRunningTime="2026-09-16 19:06:20.139984195 +0000 UTC m=+1595.664443028"
Sep 16 19:06:21.114580 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:21.114544    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:27.114177 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:27.114130    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:27.127193 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:27.127158    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:27.158870 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:27.158840    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:06:31.166541 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:31.166507    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:49.114655 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:06:49.114621    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:07:47.894612 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:07:47.894574    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:48.000690 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:07:48.000659    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:48.098400 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:07:48.098349    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:48.475369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:07:48.475334    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:49.311685 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:07:49.311645    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:52.979336 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:07:52.979294    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:54.172820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:07:54.172785    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:05.164006 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:05.163971    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:28.878695 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:28.878659    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:29.000966 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:29.000932    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:29.166109 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:29.166013    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:29.282781 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:29.282745    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:29.872335 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:29.872304    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:33.247838 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:33.247795    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:33.927459 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:33.927423    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:40.652014 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:40.651978    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:40.924007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:40.923922    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:41.047845 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:08:41.047807    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:00.692771 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:00.692732    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:03.501721 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:03.501688    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:03.621349 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:03.621321    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:03.682296 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:03.682260    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"]
Sep 16 19:09:03.683684 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:03.683652    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"]
Sep 16 19:09:03.691421 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:09:03.691387    2590 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:09:03.691421 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:03.691421 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:03.691421 ip-10-0-143-44 kubenswrapper[2590]:  > containerID="591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112" cmd=["/bin/sleep","15"]
Sep 16 19:09:03.691629 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:03.691440    2590 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww" message="" err=<
Sep 16 19:09:03.691629 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:03.691629 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:03.691629 ip-10-0-143-44 kubenswrapper[2590]:  >
Sep 16 19:09:03.691629 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:09:03.691452    2590 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:09:03.691629 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:03.691629 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:03.691629 ip-10-0-143-44 kubenswrapper[2590]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww" podUID="d9239f0c-7f62-494b-9c6f-e105b54c195b" containerName="main" containerID="cri-o://591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112"
Sep 16 19:09:03.691629 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:03.691497    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww" podUID="d9239f0c-7f62-494b-9c6f-e105b54c195b" containerName="main" containerID="cri-o://591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112" gracePeriod=60
Sep 16 19:09:03.871150 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:03.871111    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:04.057958 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.057867    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:09:04.140809 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.140773    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rprl\" (UniqueName: \"kubernetes.io/projected/d9239f0c-7f62-494b-9c6f-e105b54c195b-kube-api-access-4rprl\") pod \"d9239f0c-7f62-494b-9c6f-e105b54c195b\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") "
Sep 16 19:09:04.141031 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.140818    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-dshm\") pod \"d9239f0c-7f62-494b-9c6f-e105b54c195b\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") "
Sep 16 19:09:04.141031 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.140866    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-model-cache\") pod \"d9239f0c-7f62-494b-9c6f-e105b54c195b\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") "
Sep 16 19:09:04.141031 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.140922    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-home\") pod \"d9239f0c-7f62-494b-9c6f-e105b54c195b\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") "
Sep 16 19:09:04.141031 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.140942    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9239f0c-7f62-494b-9c6f-e105b54c195b-tls-certs\") pod \"d9239f0c-7f62-494b-9c6f-e105b54c195b\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") "
Sep 16 19:09:04.141262 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.141170    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-home" (OuterVolumeSpecName: "home") pod "d9239f0c-7f62-494b-9c6f-e105b54c195b" (UID: "d9239f0c-7f62-494b-9c6f-e105b54c195b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:04.141262 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.141182    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9239f0c-7f62-494b-9c6f-e105b54c195b" (UID: "d9239f0c-7f62-494b-9c6f-e105b54c195b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:04.143237 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.143207    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-dshm" (OuterVolumeSpecName: "dshm") pod "d9239f0c-7f62-494b-9c6f-e105b54c195b" (UID: "d9239f0c-7f62-494b-9c6f-e105b54c195b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:04.143361 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.143300    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9239f0c-7f62-494b-9c6f-e105b54c195b-kube-api-access-4rprl" (OuterVolumeSpecName: "kube-api-access-4rprl") pod "d9239f0c-7f62-494b-9c6f-e105b54c195b" (UID: "d9239f0c-7f62-494b-9c6f-e105b54c195b"). InnerVolumeSpecName "kube-api-access-4rprl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:04.143361 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.143305    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9239f0c-7f62-494b-9c6f-e105b54c195b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9239f0c-7f62-494b-9c6f-e105b54c195b" (UID: "d9239f0c-7f62-494b-9c6f-e105b54c195b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:04.241518 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.241484    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-tmp-dir\") pod \"d9239f0c-7f62-494b-9c6f-e105b54c195b\" (UID: \"d9239f0c-7f62-494b-9c6f-e105b54c195b\") "
Sep 16 19:09:04.241710 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.241695    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.241764 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.241716    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.241764 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.241730    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.241764 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.241740    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9239f0c-7f62-494b-9c6f-e105b54c195b-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.241764 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.241762    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rprl\" (UniqueName: \"kubernetes.io/projected/d9239f0c-7f62-494b-9c6f-e105b54c195b-kube-api-access-4rprl\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.241932 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.241792    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9239f0c-7f62-494b-9c6f-e105b54c195b" (UID: "d9239f0c-7f62-494b-9c6f-e105b54c195b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:04.342900 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.342796    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9239f0c-7f62-494b-9c6f-e105b54c195b-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.834474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.834438    2590 generic.go:356] "Generic (PLEG): container finished" podID="d9239f0c-7f62-494b-9c6f-e105b54c195b" containerID="591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112" exitCode=0
Sep 16 19:09:04.834474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.834476    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww" event={"ID":"d9239f0c-7f62-494b-9c6f-e105b54c195b","Type":"ContainerDied","Data":"591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112"}
Sep 16 19:09:04.834940 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.834486    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"
Sep 16 19:09:04.834940 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.834513    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww" event={"ID":"d9239f0c-7f62-494b-9c6f-e105b54c195b","Type":"ContainerDied","Data":"e4d6600b91695e6b952484414e719f40a713268b0211ced7efb504eb199c3441"}
Sep 16 19:09:04.834940 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.834537    2590 scope.go:117] "RemoveContainer" containerID="591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112"
Sep 16 19:09:04.844136 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.844119    2590 scope.go:117] "RemoveContainer" containerID="591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112"
Sep 16 19:09:04.844401 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:09:04.844379    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112\": container with ID starting with 591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112 not found: ID does not exist" containerID="591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112"
Sep 16 19:09:04.844451 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.844416    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112"} err="failed to get container status \"591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112\": rpc error: code = NotFound desc = could not find container \"591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112\": container with ID starting with 591205f4082575c096c33e4d19ddda39dddb6c2335e377c344ebab8f53197112 not found: ID does not exist"
Sep 16 19:09:04.894171 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.894136    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"]
Sep 16 19:09:04.900303 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:04.900277    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmjww"]
Sep 16 19:09:05.026604 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:05.026568    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9239f0c-7f62-494b-9c6f-e105b54c195b" path="/var/lib/kubelet/pods/d9239f0c-7f62-494b-9c6f-e105b54c195b/volumes"
Sep 16 19:09:09.115248 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:09.115207    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" podUID="ba609aac-f16c-4f11-aac0-5febc82aaebd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:11.604819 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:11.604775    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:18.502548 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:18.502500    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:18.581425 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:18.581391    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:18.702719 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:18.702670    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" podUID="ba609aac-f16c-4f11-aac0-5febc82aaebd" containerName="main" containerID="cri-o://0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2" gracePeriod=45
Sep 16 19:09:19.115307 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:19.115269    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" podUID="ba609aac-f16c-4f11-aac0-5febc82aaebd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:22.194024 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:22.193980    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:22.346949 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:22.346921    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:22.616350 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:22.616305    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:22.751921 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:22.751866    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:22.853397 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:22.853365    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:22.946782 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:22.946685    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:27.238750 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:27.238715    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:29.114286 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:29.114253    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" podUID="ba609aac-f16c-4f11-aac0-5febc82aaebd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:29.114673 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:29.114364    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:09:29.608379 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:29.608342    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:34.482748 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:34.482706    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:37.364008 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:37.363931    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:37.455542 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:37.455507    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:39.114438 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:39.114392    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" podUID="ba609aac-f16c-4f11-aac0-5febc82aaebd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:40.118085 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:40.118047    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:48.940227 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:48.940196    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:09:49.017239 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.017147    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjgz8\" (UniqueName: \"kubernetes.io/projected/ba609aac-f16c-4f11-aac0-5febc82aaebd-kube-api-access-gjgz8\") pod \"ba609aac-f16c-4f11-aac0-5febc82aaebd\" (UID: \"ba609aac-f16c-4f11-aac0-5febc82aaebd\") "
Sep 16 19:09:49.017393 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.017250    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba609aac-f16c-4f11-aac0-5febc82aaebd-tls-certs\") pod \"ba609aac-f16c-4f11-aac0-5febc82aaebd\" (UID: \"ba609aac-f16c-4f11-aac0-5febc82aaebd\") "
Sep 16 19:09:49.019386 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.019356    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba609aac-f16c-4f11-aac0-5febc82aaebd-kube-api-access-gjgz8" (OuterVolumeSpecName: "kube-api-access-gjgz8") pod "ba609aac-f16c-4f11-aac0-5febc82aaebd" (UID: "ba609aac-f16c-4f11-aac0-5febc82aaebd"). InnerVolumeSpecName "kube-api-access-gjgz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:49.019490 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.019420    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba609aac-f16c-4f11-aac0-5febc82aaebd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba609aac-f16c-4f11-aac0-5febc82aaebd" (UID: "ba609aac-f16c-4f11-aac0-5febc82aaebd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:49.031623 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.031591    2590 generic.go:356] "Generic (PLEG): container finished" podID="ba609aac-f16c-4f11-aac0-5febc82aaebd" containerID="0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2" exitCode=0
Sep 16 19:09:49.031750 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.031639    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"
Sep 16 19:09:49.031750 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.031670    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" event={"ID":"ba609aac-f16c-4f11-aac0-5febc82aaebd","Type":"ContainerDied","Data":"0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2"}
Sep 16 19:09:49.031750 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.031723    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx" event={"ID":"ba609aac-f16c-4f11-aac0-5febc82aaebd","Type":"ContainerDied","Data":"540e40863ccd0cd8a4bd8c42a240788d6709af54bd505ec93d820c5755a3f149"}
Sep 16 19:09:49.031750 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.031747    2590 scope.go:117] "RemoveContainer" containerID="0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2"
Sep 16 19:09:49.040329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.040308    2590 scope.go:117] "RemoveContainer" containerID="0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2"
Sep 16 19:09:49.040595 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:09:49.040577    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2\": container with ID starting with 0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2 not found: ID does not exist" containerID="0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2"
Sep 16 19:09:49.040648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.040605    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2"} err="failed to get container status \"0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2\": rpc error: code = NotFound desc = could not find container \"0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2\": container with ID starting with 0c323141cb8ea053310a84f2bf3bbae67e2b5fcfb3ec8550c7a16ae8c24850b2 not found: ID does not exist"
Sep 16 19:09:49.052615 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.052588    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"]
Sep 16 19:09:49.056259 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.056238    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xx2sx"]
Sep 16 19:09:49.118729 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.118698    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjgz8\" (UniqueName: \"kubernetes.io/projected/ba609aac-f16c-4f11-aac0-5febc82aaebd-kube-api-access-gjgz8\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:09:49.118729 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:49.118725    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba609aac-f16c-4f11-aac0-5febc82aaebd-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:09:50.019599 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:50.019567    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:50.206335 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:50.206299    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:51.025652 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:51.025621    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba609aac-f16c-4f11-aac0-5febc82aaebd" path="/var/lib/kubelet/pods/ba609aac-f16c-4f11-aac0-5febc82aaebd/volumes"
Sep 16 19:09:56.958562 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:56.958521    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:59.948161 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.948128    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"]
Sep 16 19:09:59.948523 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.948440    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba609aac-f16c-4f11-aac0-5febc82aaebd" containerName="main"
Sep 16 19:09:59.948523 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.948452    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba609aac-f16c-4f11-aac0-5febc82aaebd" containerName="main"
Sep 16 19:09:59.948523 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.948465    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9239f0c-7f62-494b-9c6f-e105b54c195b" containerName="main"
Sep 16 19:09:59.948523 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.948470    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9239f0c-7f62-494b-9c6f-e105b54c195b" containerName="main"
Sep 16 19:09:59.948658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.948532    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9239f0c-7f62-494b-9c6f-e105b54c195b" containerName="main"
Sep 16 19:09:59.948658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.948543    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba609aac-f16c-4f11-aac0-5febc82aaebd" containerName="main"
Sep 16 19:09:59.951234 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.951217    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:09:59.958015 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.957996    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-4kp45\""
Sep 16 19:09:59.959173 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.959149    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:09:59.965445 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.965420    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:09:59.968095 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.968074    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:09:59.982242 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.982208    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"]
Sep 16 19:09:59.992421 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:09:59.992392    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.109839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.109802    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.110069 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.109855    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210e0094-817a-42f5-bd45-5a26d911102a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.110069 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.109914    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.110069 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.109936    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.110069 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.109980    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.110069 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.110004    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnrd\" (UniqueName: \"kubernetes.io/projected/210e0094-817a-42f5-bd45-5a26d911102a-kube-api-access-qrnrd\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.210753 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.210651    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210e0094-817a-42f5-bd45-5a26d911102a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.210753 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.210704    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.211042 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.210827    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.211042 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.210915    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.211042 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.210945    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnrd\" (UniqueName: \"kubernetes.io/projected/210e0094-817a-42f5-bd45-5a26d911102a-kube-api-access-qrnrd\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.211042 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.210980    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.211245 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.211094    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.211294 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.211252    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.211333 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.211306    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.213320 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.213298    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.213543 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.213523    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210e0094-817a-42f5-bd45-5a26d911102a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.228071 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.228047    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnrd\" (UniqueName: \"kubernetes.io/projected/210e0094-817a-42f5-bd45-5a26d911102a-kube-api-access-qrnrd\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.261934 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.261866    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:00.290947 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.290916    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"]
Sep 16 19:10:00.294505 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.294480    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:00.297115 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.297092    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-zhrqv\""
Sep 16 19:10:00.307924 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.307866    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"]
Sep 16 19:10:00.338666 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.338635    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.413289 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.413248    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76b1f5f-dce8-4840-b82e-ad72490d86f0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedc75bf\" (UID: \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:00.413477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.413356    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqr9g\" (UniqueName: \"kubernetes.io/projected/c76b1f5f-dce8-4840-b82e-ad72490d86f0-kube-api-access-wqr9g\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedc75bf\" (UID: \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:00.514707 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.514609    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqr9g\" (UniqueName: \"kubernetes.io/projected/c76b1f5f-dce8-4840-b82e-ad72490d86f0-kube-api-access-wqr9g\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedc75bf\" (UID: \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:00.514707 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.514669    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76b1f5f-dce8-4840-b82e-ad72490d86f0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedc75bf\" (UID: \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:00.517390 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.517369    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76b1f5f-dce8-4840-b82e-ad72490d86f0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedc75bf\" (UID: \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:00.527278 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.527250    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqr9g\" (UniqueName: \"kubernetes.io/projected/c76b1f5f-dce8-4840-b82e-ad72490d86f0-kube-api-access-wqr9g\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedc75bf\" (UID: \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:00.546730 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.546699    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"]
Sep 16 19:10:00.549581 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:10:00.549558    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210e0094_817a_42f5_bd45_5a26d911102a.slice/crio-2e24cfe6c1663b3cb19a6f6e5882d7929866cde10130622c588e8eb39fd9bb4f WatchSource:0}: Error finding container 2e24cfe6c1663b3cb19a6f6e5882d7929866cde10130622c588e8eb39fd9bb4f: Status 404 returned error can't find the container with id 2e24cfe6c1663b3cb19a6f6e5882d7929866cde10130622c588e8eb39fd9bb4f
Sep 16 19:10:00.606200 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.606178    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:00.898557 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:00.898532    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"]
Sep 16 19:10:00.900573 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:10:00.900541    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76b1f5f_dce8_4840_b82e_ad72490d86f0.slice/crio-911fac07b479d3d322011d47d5a42690237a7124460e412db4cac38f9c3854bc WatchSource:0}: Error finding container 911fac07b479d3d322011d47d5a42690237a7124460e412db4cac38f9c3854bc: Status 404 returned error can't find the container with id 911fac07b479d3d322011d47d5a42690237a7124460e412db4cac38f9c3854bc
Sep 16 19:10:01.084148 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:01.083817    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" event={"ID":"c76b1f5f-dce8-4840-b82e-ad72490d86f0","Type":"ContainerStarted","Data":"4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0"}
Sep 16 19:10:01.084643 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:01.084199    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" event={"ID":"c76b1f5f-dce8-4840-b82e-ad72490d86f0","Type":"ContainerStarted","Data":"911fac07b479d3d322011d47d5a42690237a7124460e412db4cac38f9c3854bc"}
Sep 16 19:10:01.084750 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:01.084728    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:01.086165 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:01.086134    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25" event={"ID":"210e0094-817a-42f5-bd45-5a26d911102a","Type":"ContainerStarted","Data":"a333825da1923387d9040a4fa1f0cd922697c61ec1a879e355b3fe1ff1233b84"}
Sep 16 19:10:01.086317 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:01.086184    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25" event={"ID":"210e0094-817a-42f5-bd45-5a26d911102a","Type":"ContainerStarted","Data":"2e24cfe6c1663b3cb19a6f6e5882d7929866cde10130622c588e8eb39fd9bb4f"}
Sep 16 19:10:01.111206 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:01.111109    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" podStartSLOduration=1.111090179 podStartE2EDuration="1.111090179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:01.109863563 +0000 UTC m=+1816.634322399" watchObservedRunningTime="2026-09-16 19:10:01.111090179 +0000 UTC m=+1816.635549013"
Sep 16 19:10:01.141152 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:01.141097    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25" podStartSLOduration=2.141076093 podStartE2EDuration="2.141076093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:01.137432331 +0000 UTC m=+1816.661891167" watchObservedRunningTime="2026-09-16 19:10:01.141076093 +0000 UTC m=+1816.665534927"
Sep 16 19:10:01.262946 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:01.262829    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:03.844279 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:03.844232    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:04.023112 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:04.023075    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:05.826543 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:05.826501    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:10.262152 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:10.262106    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:10.274843 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:10.274812    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:11.143142 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:11.143115    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:12.828002 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:12.827961    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:12.943601 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:12.943560    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:13.396615 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:13.396581    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:24.500103 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:24.500058    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:26.237208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:26.237178    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:33.100216 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:33.100183    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:10:43.730648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:43.730611    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"]
Sep 16 19:10:43.734509 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:43.734461    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:43.738965 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:43.738937    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"]
Sep 16 19:10:43.741549 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:10:43.741511    2590 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:10:43.741549 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:10:43.741549 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:10:43.741549 ip-10-0-143-44 kubenswrapper[2590]:  > containerID="a333825da1923387d9040a4fa1f0cd922697c61ec1a879e355b3fe1ff1233b84" cmd=["/bin/sleep","15"]
Sep 16 19:10:43.741807 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:43.741561    2590 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25" message="" err=<
Sep 16 19:10:43.741807 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:10:43.741807 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:10:43.741807 ip-10-0-143-44 kubenswrapper[2590]:  >
Sep 16 19:10:43.741807 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:10:43.741575    2590 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:10:43.741807 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:10:43.741807 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:10:43.741807 ip-10-0-143-44 kubenswrapper[2590]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25" podUID="210e0094-817a-42f5-bd45-5a26d911102a" containerName="main" containerID="cri-o://a333825da1923387d9040a4fa1f0cd922697c61ec1a879e355b3fe1ff1233b84"
Sep 16 19:10:43.741807 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:43.741612    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25" podUID="210e0094-817a-42f5-bd45-5a26d911102a" containerName="main" containerID="cri-o://a333825da1923387d9040a4fa1f0cd922697c61ec1a879e355b3fe1ff1233b84" gracePeriod=60
Sep 16 19:10:43.826821 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:43.826787    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:44.133094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.133054    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25" podUID="210e0094-817a-42f5-bd45-5a26d911102a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 16 19:10:44.276263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.276229    2590 generic.go:356] "Generic (PLEG): container finished" podID="210e0094-817a-42f5-bd45-5a26d911102a" containerID="a333825da1923387d9040a4fa1f0cd922697c61ec1a879e355b3fe1ff1233b84" exitCode=0
Sep 16 19:10:44.276426 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.276307    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25" event={"ID":"210e0094-817a-42f5-bd45-5a26d911102a","Type":"ContainerDied","Data":"a333825da1923387d9040a4fa1f0cd922697c61ec1a879e355b3fe1ff1233b84"}
Sep 16 19:10:44.492320 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.492295    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:44.589510 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.589475    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210e0094-817a-42f5-bd45-5a26d911102a-tls-certs\") pod \"210e0094-817a-42f5-bd45-5a26d911102a\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") "
Sep 16 19:10:44.589713 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.589560    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-dshm\") pod \"210e0094-817a-42f5-bd45-5a26d911102a\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") "
Sep 16 19:10:44.589713 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.589603    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-home\") pod \"210e0094-817a-42f5-bd45-5a26d911102a\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") "
Sep 16 19:10:44.589713 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.589621    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-model-cache\") pod \"210e0094-817a-42f5-bd45-5a26d911102a\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") "
Sep 16 19:10:44.589713 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.589639    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-tmp-dir\") pod \"210e0094-817a-42f5-bd45-5a26d911102a\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") "
Sep 16 19:10:44.589713 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.589674    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrnrd\" (UniqueName: \"kubernetes.io/projected/210e0094-817a-42f5-bd45-5a26d911102a-kube-api-access-qrnrd\") pod \"210e0094-817a-42f5-bd45-5a26d911102a\" (UID: \"210e0094-817a-42f5-bd45-5a26d911102a\") "
Sep 16 19:10:44.590053 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.589955    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-model-cache" (OuterVolumeSpecName: "model-cache") pod "210e0094-817a-42f5-bd45-5a26d911102a" (UID: "210e0094-817a-42f5-bd45-5a26d911102a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:44.590053 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.589942    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-home" (OuterVolumeSpecName: "home") pod "210e0094-817a-42f5-bd45-5a26d911102a" (UID: "210e0094-817a-42f5-bd45-5a26d911102a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:44.590053 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.589984    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "210e0094-817a-42f5-bd45-5a26d911102a" (UID: "210e0094-817a-42f5-bd45-5a26d911102a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:44.592076 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.592052    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-dshm" (OuterVolumeSpecName: "dshm") pod "210e0094-817a-42f5-bd45-5a26d911102a" (UID: "210e0094-817a-42f5-bd45-5a26d911102a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:44.592076 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.592057    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210e0094-817a-42f5-bd45-5a26d911102a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "210e0094-817a-42f5-bd45-5a26d911102a" (UID: "210e0094-817a-42f5-bd45-5a26d911102a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:44.592245 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.592108    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210e0094-817a-42f5-bd45-5a26d911102a-kube-api-access-qrnrd" (OuterVolumeSpecName: "kube-api-access-qrnrd") pod "210e0094-817a-42f5-bd45-5a26d911102a" (UID: "210e0094-817a-42f5-bd45-5a26d911102a"). InnerVolumeSpecName "kube-api-access-qrnrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:44.690779 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.690663    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.690779 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.690703    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.690779 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.690715    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.690779 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.690727    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210e0094-817a-42f5-bd45-5a26d911102a-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.690779 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.690740    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrnrd\" (UniqueName: \"kubernetes.io/projected/210e0094-817a-42f5-bd45-5a26d911102a-kube-api-access-qrnrd\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.690779 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:44.690751    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210e0094-817a-42f5-bd45-5a26d911102a-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:10:45.282210 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:45.282173    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25" event={"ID":"210e0094-817a-42f5-bd45-5a26d911102a","Type":"ContainerDied","Data":"2e24cfe6c1663b3cb19a6f6e5882d7929866cde10130622c588e8eb39fd9bb4f"}
Sep 16 19:10:45.282210 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:45.282202    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"
Sep 16 19:10:45.282713 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:45.282227    2590 scope.go:117] "RemoveContainer" containerID="a333825da1923387d9040a4fa1f0cd922697c61ec1a879e355b3fe1ff1233b84"
Sep 16 19:10:45.304279 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:45.304244    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"]
Sep 16 19:10:45.310117 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:45.310090    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444msf25"]
Sep 16 19:10:47.025038 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:47.025001    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="210e0094-817a-42f5-bd45-5a26d911102a" path="/var/lib/kubelet/pods/210e0094-817a-42f5-bd45-5a26d911102a/volumes"
Sep 16 19:10:47.915858 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:47.915822    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:48.940266 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:48.940235    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:49.051082 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:49.051046    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:49.166752 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:49.166721    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:53.100139 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:53.100093    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" podUID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:10:54.013638 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:54.013604    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:55.486852 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:55.486811    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:58.757680 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:10:58.757640    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" podUID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" containerName="main" containerID="cri-o://4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0" gracePeriod=45
Sep 16 19:11:03.100765 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:03.100728    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" podUID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:03.676871 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:03.676835    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:09.569239 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:09.569152    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:12.213239 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:12.213201    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:13.100738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:13.100704    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" podUID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:13.100929 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:13.100802    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:11:15.108528 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:15.108493    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:18.869219 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:18.869175    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:20.534581 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:20.534550    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:23.100482 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:23.100438    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" podUID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:29.006097 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.006068    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:11:29.060484 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.060452    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76b1f5f-dce8-4840-b82e-ad72490d86f0-tls-certs\") pod \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\" (UID: \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\") "
Sep 16 19:11:29.060665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.060525    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqr9g\" (UniqueName: \"kubernetes.io/projected/c76b1f5f-dce8-4840-b82e-ad72490d86f0-kube-api-access-wqr9g\") pod \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\" (UID: \"c76b1f5f-dce8-4840-b82e-ad72490d86f0\") "
Sep 16 19:11:29.062788 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.062757    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76b1f5f-dce8-4840-b82e-ad72490d86f0-kube-api-access-wqr9g" (OuterVolumeSpecName: "kube-api-access-wqr9g") pod "c76b1f5f-dce8-4840-b82e-ad72490d86f0" (UID: "c76b1f5f-dce8-4840-b82e-ad72490d86f0"). InnerVolumeSpecName "kube-api-access-wqr9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:29.062894 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.062855    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76b1f5f-dce8-4840-b82e-ad72490d86f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c76b1f5f-dce8-4840-b82e-ad72490d86f0" (UID: "c76b1f5f-dce8-4840-b82e-ad72490d86f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:29.161312 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.161274    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqr9g\" (UniqueName: \"kubernetes.io/projected/c76b1f5f-dce8-4840-b82e-ad72490d86f0-kube-api-access-wqr9g\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:29.161312 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.161304    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76b1f5f-dce8-4840-b82e-ad72490d86f0-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:29.472665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.472569    2590 generic.go:356] "Generic (PLEG): container finished" podID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" containerID="4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0" exitCode=0
Sep 16 19:11:29.472665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.472610    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"
Sep 16 19:11:29.472665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.472646    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" event={"ID":"c76b1f5f-dce8-4840-b82e-ad72490d86f0","Type":"ContainerDied","Data":"4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0"}
Sep 16 19:11:29.472937 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.472692    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf" event={"ID":"c76b1f5f-dce8-4840-b82e-ad72490d86f0","Type":"ContainerDied","Data":"911fac07b479d3d322011d47d5a42690237a7124460e412db4cac38f9c3854bc"}
Sep 16 19:11:29.472937 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.472717    2590 scope.go:117] "RemoveContainer" containerID="4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0"
Sep 16 19:11:29.481815 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.481796    2590 scope.go:117] "RemoveContainer" containerID="4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0"
Sep 16 19:11:29.482152 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:11:29.482133    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0\": container with ID starting with 4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0 not found: ID does not exist" containerID="4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0"
Sep 16 19:11:29.482217 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.482161    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0"} err="failed to get container status \"4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0\": rpc error: code = NotFound desc = could not find container \"4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0\": container with ID starting with 4c9840d2ade925c66d16d88499aa77ba505889a0692f829b64320302e6950df0 not found: ID does not exist"
Sep 16 19:11:29.496581 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.496551    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"]
Sep 16 19:11:29.505189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:29.505166    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedc75bf"]
Sep 16 19:11:31.025820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:31.025775    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" path="/var/lib/kubelet/pods/c76b1f5f-dce8-4840-b82e-ad72490d86f0/volumes"
Sep 16 19:11:32.398069 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:32.398031    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:32.485993 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:32.485958    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:38.578662 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.578630    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"]
Sep 16 19:11:38.579043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.578914    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="210e0094-817a-42f5-bd45-5a26d911102a" containerName="main"
Sep 16 19:11:38.579043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.578925    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="210e0094-817a-42f5-bd45-5a26d911102a" containerName="main"
Sep 16 19:11:38.579043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.578936    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" containerName="main"
Sep 16 19:11:38.579043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.578941    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" containerName="main"
Sep 16 19:11:38.579043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.578996    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="210e0094-817a-42f5-bd45-5a26d911102a" containerName="main"
Sep 16 19:11:38.579043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.579007    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76b1f5f-dce8-4840-b82e-ad72490d86f0" containerName="main"
Sep 16 19:11:38.583224 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.583207    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.586948 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.586922    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:11:38.587075 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.587004    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:11:38.587946 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.587911    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-frc9h\""
Sep 16 19:11:38.588064 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.587944    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:11:38.588064 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.587911    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:11:38.588064 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.588020    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:11:38.606303 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.606273    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"]
Sep 16 19:11:38.617500 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.617471    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:38.745705 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.745668    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-model-cache\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.745705 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.745710    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.745991 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.745725    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3f75ee-9480-43fa-949b-982008c54017-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.745991 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.745775    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea3f75ee-9480-43fa-949b-982008c54017-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.745991 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.745819    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-dshm\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.745991 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.745841    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-home\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.745991 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.745910    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.745991 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.745937    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29n54\" (UniqueName: \"kubernetes.io/projected/ea3f75ee-9480-43fa-949b-982008c54017-kube-api-access-29n54\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847110 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847014    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-dshm\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847110 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847056    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-home\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847110 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847089    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847110 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847111    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29n54\" (UniqueName: \"kubernetes.io/projected/ea3f75ee-9480-43fa-949b-982008c54017-kube-api-access-29n54\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847441 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847142    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-model-cache\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847441 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847161    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847441 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847177    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3f75ee-9480-43fa-949b-982008c54017-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847441 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847194    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea3f75ee-9480-43fa-949b-982008c54017-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847602 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847510    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847602 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847544    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-home\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847708 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847687    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-model-cache\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.847745 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.847712    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.848058 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.848040    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea3f75ee-9480-43fa-949b-982008c54017-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.849871 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.849839    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-dshm\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.850015 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.849978    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3f75ee-9480-43fa-949b-982008c54017-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.863227 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.863196    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29n54\" (UniqueName: \"kubernetes.io/projected/ea3f75ee-9480-43fa-949b-982008c54017-kube-api-access-29n54\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.894106 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.894073    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:38.910416 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.910382    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"]
Sep 16 19:11:38.913858 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.913833    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:38.943051 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.943020    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"]
Sep 16 19:11:38.965985 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:38.965903    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:39.049685 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.049646    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a59fbc-b62c-4187-8314-b18907eaba84-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.049866 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.049714    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50a59fbc-b62c-4187-8314-b18907eaba84-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.049866 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.049760    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.049866 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.049796    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.114057 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.113968    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:39.151097 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.151058    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.151312 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.151111    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.151312 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.151184    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a59fbc-b62c-4187-8314-b18907eaba84-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.151312 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.151229    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50a59fbc-b62c-4187-8314-b18907eaba84-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.151572 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.151546    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.151642 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.151608    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.151931 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.151910    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50a59fbc-b62c-4187-8314-b18907eaba84-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.154019 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.153999    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a59fbc-b62c-4187-8314-b18907eaba84-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.188306 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.188283    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"]
Sep 16 19:11:39.189713 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:11:39.189681    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3f75ee_9480_43fa_949b_982008c54017.slice/crio-80f0f67773d70360838f14535cf669e2b72b74fb17161b2bb14cff989b10e7c1 WatchSource:0}: Error finding container 80f0f67773d70360838f14535cf669e2b72b74fb17161b2bb14cff989b10e7c1: Status 404 returned error can't find the container with id 80f0f67773d70360838f14535cf669e2b72b74fb17161b2bb14cff989b10e7c1
Sep 16 19:11:39.229214 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.229188    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:11:39.517010 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.516978    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"]
Sep 16 19:11:39.518949 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:11:39.518917    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a59fbc_b62c_4187_8314_b18907eaba84.slice/crio-7e0d49d23007e0333a681e009ce33563c42c0fd981a59b8809e4f659b223b660 WatchSource:0}: Error finding container 7e0d49d23007e0333a681e009ce33563c42c0fd981a59b8809e4f659b223b660: Status 404 returned error can't find the container with id 7e0d49d23007e0333a681e009ce33563c42c0fd981a59b8809e4f659b223b660
Sep 16 19:11:39.519427 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.519406    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" event={"ID":"ea3f75ee-9480-43fa-949b-982008c54017","Type":"ContainerStarted","Data":"960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12"}
Sep 16 19:11:39.519532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:39.519454    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" event={"ID":"ea3f75ee-9480-43fa-949b-982008c54017","Type":"ContainerStarted","Data":"80f0f67773d70360838f14535cf669e2b72b74fb17161b2bb14cff989b10e7c1"}
Sep 16 19:11:40.529333 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:40.529284    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc" event={"ID":"50a59fbc-b62c-4187-8314-b18907eaba84","Type":"ContainerStarted","Data":"1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc"}
Sep 16 19:11:40.529785 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:40.529402    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc" event={"ID":"50a59fbc-b62c-4187-8314-b18907eaba84","Type":"ContainerStarted","Data":"7e0d49d23007e0333a681e009ce33563c42c0fd981a59b8809e4f659b223b660"}
Sep 16 19:11:40.845576 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:40.845497    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:42.543074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:42.543041    2590 generic.go:356] "Generic (PLEG): container finished" podID="ea3f75ee-9480-43fa-949b-982008c54017" containerID="960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12" exitCode=0
Sep 16 19:11:42.543518 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:42.543117    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" event={"ID":"ea3f75ee-9480-43fa-949b-982008c54017","Type":"ContainerDied","Data":"960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12"}
Sep 16 19:11:42.544891 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:42.544854    2590 generic.go:356] "Generic (PLEG): container finished" podID="50a59fbc-b62c-4187-8314-b18907eaba84" containerID="1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc" exitCode=0
Sep 16 19:11:42.544985 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:42.544905    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc" event={"ID":"50a59fbc-b62c-4187-8314-b18907eaba84","Type":"ContainerDied","Data":"1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc"}
Sep 16 19:11:42.546033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:42.546017    2590 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:11:43.551551 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:43.551516    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" event={"ID":"ea3f75ee-9480-43fa-949b-982008c54017","Type":"ContainerStarted","Data":"7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854"}
Sep 16 19:11:43.579845 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:43.579787    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" podStartSLOduration=2.940922412 podStartE2EDuration="5.579768824s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.638846412s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:43.578542362 +0000 UTC m=+1919.103001196" watchObservedRunningTime="2026-09-16 19:11:43.579768824 +0000 UTC m=+1919.104227658"
Sep 16 19:11:43.894845 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:43.894750    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:47.702231 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:47.702192    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:47.799951 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:47.799909    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:48.432154 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:48.432087    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:48.894600 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:48.894559    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:48.913290 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:48.913257    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:49.605915 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:49.605865    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:11:50.915380 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:11:50.915341    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:06.694506 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:06.694465    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:06.821101 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:06.821066    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:06.914643 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:06.914603    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:11.759148 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:11.759109    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:12.807922 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:12.807871    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:18.779279 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:18.779240    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:30.821535 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:30.821498    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc" event={"ID":"50a59fbc-b62c-4187-8314-b18907eaba84","Type":"ContainerStarted","Data":"8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd"}
Sep 16 19:12:30.822870 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:30.822838    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:12:30.846287 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:30.846216    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc" podStartSLOduration=2.454302352 podStartE2EDuration="52.846193867s" totalImagesPullingTime="47.777975048s" totalInitContainerRuntime="2.613916467s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:30.844806605 +0000 UTC m=+1966.369265445" watchObservedRunningTime="2026-09-16 19:12:30.846193867 +0000 UTC m=+1966.370652708"
Sep 16 19:12:49.623642 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:49.623602    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:56.102064 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:56.102032    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:56.210056 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:12:56.210020    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:01.837604 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:01.837569    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:13:01.876613 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:01.876532    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:01.987225 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:01.987193    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:02.666277 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:02.666189    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:04.950849 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:04.950821    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc_50a59fbc-b62c-4187-8314-b18907eaba84/main/0.log"
Sep 16 19:13:06.846391 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:06.846340    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:08.654934 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:08.654907    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:20.083816 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:20.083777    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:36.597395 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:36.597357    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:36.611755 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:36.611723    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"]
Sep 16 19:13:36.612031 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:36.612004    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc" podUID="50a59fbc-b62c-4187-8314-b18907eaba84" containerName="main" containerID="cri-o://8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd" gracePeriod=30
Sep 16 19:13:36.623688 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:36.623656    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"]
Sep 16 19:13:36.633065 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:13:36.633023    2590 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:13:36.633065 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.633065 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.633065 ip-10-0-143-44 kubenswrapper[2590]:  > containerID="7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854" cmd=["/bin/sleep","15"]
Sep 16 19:13:36.633366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:36.633080    2590 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" message="" err=<
Sep 16 19:13:36.633366 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.633366 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.633366 ip-10-0-143-44 kubenswrapper[2590]:  >
Sep 16 19:13:36.633366 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:13:36.633092    2590 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:13:36.633366 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.633366 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.633366 ip-10-0-143-44 kubenswrapper[2590]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" podUID="ea3f75ee-9480-43fa-949b-982008c54017" containerName="main" containerID="cri-o://7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854"
Sep 16 19:13:36.633366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:36.633141    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" podUID="ea3f75ee-9480-43fa-949b-982008c54017" containerName="main" containerID="cri-o://7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854" gracePeriod=60
Sep 16 19:13:36.668689 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:36.668661    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:36.990529 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:36.990495    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:13:37.082087 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082049    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-dshm\") pod \"ea3f75ee-9480-43fa-949b-982008c54017\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") "
Sep 16 19:13:37.082252 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082112    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-home\") pod \"ea3f75ee-9480-43fa-949b-982008c54017\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") "
Sep 16 19:13:37.082252 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082143    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-tmp-dir\") pod \"ea3f75ee-9480-43fa-949b-982008c54017\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") "
Sep 16 19:13:37.082252 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082203    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-model-cache\") pod \"ea3f75ee-9480-43fa-949b-982008c54017\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") "
Sep 16 19:13:37.082252 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082246    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea3f75ee-9480-43fa-949b-982008c54017-cabundle-cert\") pod \"ea3f75ee-9480-43fa-949b-982008c54017\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") "
Sep 16 19:13:37.082461 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082273    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-kserve-provision-location\") pod \"ea3f75ee-9480-43fa-949b-982008c54017\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") "
Sep 16 19:13:37.082461 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082303    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3f75ee-9480-43fa-949b-982008c54017-tls-certs\") pod \"ea3f75ee-9480-43fa-949b-982008c54017\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") "
Sep 16 19:13:37.082461 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082353    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29n54\" (UniqueName: \"kubernetes.io/projected/ea3f75ee-9480-43fa-949b-982008c54017-kube-api-access-29n54\") pod \"ea3f75ee-9480-43fa-949b-982008c54017\" (UID: \"ea3f75ee-9480-43fa-949b-982008c54017\") "
Sep 16 19:13:37.082461 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082378    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-home" (OuterVolumeSpecName: "home") pod "ea3f75ee-9480-43fa-949b-982008c54017" (UID: "ea3f75ee-9480-43fa-949b-982008c54017"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.082675 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082514    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea3f75ee-9480-43fa-949b-982008c54017" (UID: "ea3f75ee-9480-43fa-949b-982008c54017"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.082675 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082598    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea3f75ee-9480-43fa-949b-982008c54017" (UID: "ea3f75ee-9480-43fa-949b-982008c54017"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.082675 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082603    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.082675 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.082675    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.083084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.083048    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3f75ee-9480-43fa-949b-982008c54017-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea3f75ee-9480-43fa-949b-982008c54017" (UID: "ea3f75ee-9480-43fa-949b-982008c54017"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:37.084572 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.084544    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-dshm" (OuterVolumeSpecName: "dshm") pod "ea3f75ee-9480-43fa-949b-982008c54017" (UID: "ea3f75ee-9480-43fa-949b-982008c54017"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.085033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.085006    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3f75ee-9480-43fa-949b-982008c54017-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea3f75ee-9480-43fa-949b-982008c54017" (UID: "ea3f75ee-9480-43fa-949b-982008c54017"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:37.085169 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.085105    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3f75ee-9480-43fa-949b-982008c54017-kube-api-access-29n54" (OuterVolumeSpecName: "kube-api-access-29n54") pod "ea3f75ee-9480-43fa-949b-982008c54017" (UID: "ea3f75ee-9480-43fa-949b-982008c54017"). InnerVolumeSpecName "kube-api-access-29n54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:37.105161 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.105080    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea3f75ee-9480-43fa-949b-982008c54017" (UID: "ea3f75ee-9480-43fa-949b-982008c54017"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.113406 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.113379    2590 generic.go:356] "Generic (PLEG): container finished" podID="ea3f75ee-9480-43fa-949b-982008c54017" containerID="7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854" exitCode=0
Sep 16 19:13:37.113513 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.113433    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"
Sep 16 19:13:37.113513 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.113431    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" event={"ID":"ea3f75ee-9480-43fa-949b-982008c54017","Type":"ContainerDied","Data":"7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854"}
Sep 16 19:13:37.113584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.113553    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn" event={"ID":"ea3f75ee-9480-43fa-949b-982008c54017","Type":"ContainerDied","Data":"80f0f67773d70360838f14535cf669e2b72b74fb17161b2bb14cff989b10e7c1"}
Sep 16 19:13:37.113584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.113576    2590 scope.go:117] "RemoveContainer" containerID="7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854"
Sep 16 19:13:37.124287 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.124263    2590 scope.go:117] "RemoveContainer" containerID="960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12"
Sep 16 19:13:37.133294 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.133273    2590 scope.go:117] "RemoveContainer" containerID="7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854"
Sep 16 19:13:37.133595 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:13:37.133571    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854\": container with ID starting with 7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854 not found: ID does not exist" containerID="7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854"
Sep 16 19:13:37.133692 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.133603    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854"} err="failed to get container status \"7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854\": rpc error: code = NotFound desc = could not find container \"7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854\": container with ID starting with 7d7707e2e3c8db930e3f83da8e7c7f4624dd86b27553e15932ca9d6f464ac854 not found: ID does not exist"
Sep 16 19:13:37.133692 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.133621    2590 scope.go:117] "RemoveContainer" containerID="960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12"
Sep 16 19:13:37.133991 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:13:37.133972    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12\": container with ID starting with 960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12 not found: ID does not exist" containerID="960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12"
Sep 16 19:13:37.134060 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.134000    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12"} err="failed to get container status \"960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12\": rpc error: code = NotFound desc = could not find container \"960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12\": container with ID starting with 960fc91a1612467ab8b999a73e62f57d833949b72c318f97d161a2eca89e8e12 not found: ID does not exist"
Sep 16 19:13:37.138751 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.138725    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"]
Sep 16 19:13:37.142985 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.142963    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-nc9cn"]
Sep 16 19:13:37.183316 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.183287    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.183435 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.183322    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea3f75ee-9480-43fa-949b-982008c54017-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.183435 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.183334    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.183435 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.183348    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3f75ee-9480-43fa-949b-982008c54017-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.183435 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.183360    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29n54\" (UniqueName: \"kubernetes.io/projected/ea3f75ee-9480-43fa-949b-982008c54017-kube-api-access-29n54\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.183435 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.183367    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3f75ee-9480-43fa-949b-982008c54017-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.253356 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.253327    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:13:37.385049 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.384942    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a59fbc-b62c-4187-8314-b18907eaba84-tls-certs\") pod \"50a59fbc-b62c-4187-8314-b18907eaba84\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") "
Sep 16 19:13:37.385049 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.384998    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-tmp-dir\") pod \"50a59fbc-b62c-4187-8314-b18907eaba84\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") "
Sep 16 19:13:37.385049 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.385034    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50a59fbc-b62c-4187-8314-b18907eaba84-cabundle-cert\") pod \"50a59fbc-b62c-4187-8314-b18907eaba84\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") "
Sep 16 19:13:37.385362 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.385075    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-kserve-provision-location\") pod \"50a59fbc-b62c-4187-8314-b18907eaba84\" (UID: \"50a59fbc-b62c-4187-8314-b18907eaba84\") "
Sep 16 19:13:37.385432 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.385367    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a59fbc-b62c-4187-8314-b18907eaba84-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50a59fbc-b62c-4187-8314-b18907eaba84" (UID: "50a59fbc-b62c-4187-8314-b18907eaba84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:37.385549 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.385482    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50a59fbc-b62c-4187-8314-b18907eaba84" (UID: "50a59fbc-b62c-4187-8314-b18907eaba84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.387443 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.387419    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a59fbc-b62c-4187-8314-b18907eaba84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50a59fbc-b62c-4187-8314-b18907eaba84" (UID: "50a59fbc-b62c-4187-8314-b18907eaba84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:37.406119 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.406074    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50a59fbc-b62c-4187-8314-b18907eaba84" (UID: "50a59fbc-b62c-4187-8314-b18907eaba84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.486225 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.486174    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.486225 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.486208    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a59fbc-b62c-4187-8314-b18907eaba84-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.486225 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.486219    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50a59fbc-b62c-4187-8314-b18907eaba84-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.486225 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:37.486231    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50a59fbc-b62c-4187-8314-b18907eaba84-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:38.120857 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.120822    2590 generic.go:356] "Generic (PLEG): container finished" podID="50a59fbc-b62c-4187-8314-b18907eaba84" containerID="8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd" exitCode=0
Sep 16 19:13:38.121278 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.120872    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"
Sep 16 19:13:38.121278 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.120909    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc" event={"ID":"50a59fbc-b62c-4187-8314-b18907eaba84","Type":"ContainerDied","Data":"8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd"}
Sep 16 19:13:38.121278 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.120964    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc" event={"ID":"50a59fbc-b62c-4187-8314-b18907eaba84","Type":"ContainerDied","Data":"7e0d49d23007e0333a681e009ce33563c42c0fd981a59b8809e4f659b223b660"}
Sep 16 19:13:38.121278 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.120987    2590 scope.go:117] "RemoveContainer" containerID="8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd"
Sep 16 19:13:38.129903 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.129866    2590 scope.go:117] "RemoveContainer" containerID="1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc"
Sep 16 19:13:38.140277 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.140260    2590 scope.go:117] "RemoveContainer" containerID="8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd"
Sep 16 19:13:38.140577 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:13:38.140556    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd\": container with ID starting with 8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd not found: ID does not exist" containerID="8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd"
Sep 16 19:13:38.140628 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.140585    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd"} err="failed to get container status \"8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd\": rpc error: code = NotFound desc = could not find container \"8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd\": container with ID starting with 8e5fa6aa34a05c7e1a8e79b85359170d2879338a5b32ee62c52b0258a30deefd not found: ID does not exist"
Sep 16 19:13:38.140628 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.140602    2590 scope.go:117] "RemoveContainer" containerID="1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc"
Sep 16 19:13:38.140951 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:13:38.140929    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc\": container with ID starting with 1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc not found: ID does not exist" containerID="1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc"
Sep 16 19:13:38.141023 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.140961    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc"} err="failed to get container status \"1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc\": rpc error: code = NotFound desc = could not find container \"1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc\": container with ID starting with 1d817be328680c1c6cd0108f4cc8e439ac36b5b4d7e189393cbb7a97356f2bdc not found: ID does not exist"
Sep 16 19:13:38.148928 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.148899    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"]
Sep 16 19:13:38.153211 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:38.153188    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7879fbd678-tmqbc"]
Sep 16 19:13:39.025547 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:39.025516    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50a59fbc-b62c-4187-8314-b18907eaba84" path="/var/lib/kubelet/pods/50a59fbc-b62c-4187-8314-b18907eaba84/volumes"
Sep 16 19:13:39.026000 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:39.025985    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea3f75ee-9480-43fa-949b-982008c54017" path="/var/lib/kubelet/pods/ea3f75ee-9480-43fa-949b-982008c54017/volumes"
Sep 16 19:13:41.030826 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:41.030794    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:44.287029 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:13:44.286990    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:20.516981 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:20.516872    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:25.530592 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:25.530556    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:25.635048 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:25.635005    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:32.544194 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:32.544153    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:38.525242 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:38.525206    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:38.602463 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:38.602415    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:51.267626 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267591    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"]
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267868    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3f75ee-9480-43fa-949b-982008c54017" containerName="storage-initializer"
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267892    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3f75ee-9480-43fa-949b-982008c54017" containerName="storage-initializer"
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267903    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3f75ee-9480-43fa-949b-982008c54017" containerName="main"
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267908    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3f75ee-9480-43fa-949b-982008c54017" containerName="main"
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267915    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a59fbc-b62c-4187-8314-b18907eaba84" containerName="main"
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267920    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a59fbc-b62c-4187-8314-b18907eaba84" containerName="main"
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267927    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a59fbc-b62c-4187-8314-b18907eaba84" containerName="storage-initializer"
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267933    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a59fbc-b62c-4187-8314-b18907eaba84" containerName="storage-initializer"
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267986    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea3f75ee-9480-43fa-949b-982008c54017" containerName="main"
Sep 16 19:14:51.268065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.267993    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a59fbc-b62c-4187-8314-b18907eaba84" containerName="main"
Sep 16 19:14:51.270290 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.270272    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.273005 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.272984    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:51.273739 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.273722    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:14:51.273820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.273722    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:14:51.273820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.273814    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:51.273986 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.273831    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-gn7n2\""
Sep 16 19:14:51.273986 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.273910    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:14:51.274161 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.274142    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-home\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.274222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.274168    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-dshm\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.274222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.274193    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tls-certs\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.274329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.274260    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.274329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.274294    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.274329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.274322    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.274475 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.274388    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrfs\" (UniqueName: \"kubernetes.io/projected/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kube-api-access-qwrfs\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.274475 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.274409    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-model-cache\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.297532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.297495    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"]
Sep 16 19:14:51.333352 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.333302    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:51.375669 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.375631    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-home\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.375669 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.375665    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-dshm\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.375977 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.375685    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tls-certs\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.375977 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.375801    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.375977 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.375842    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.375977 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.375872    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.375977 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.375946    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrfs\" (UniqueName: \"kubernetes.io/projected/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kube-api-access-qwrfs\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.375977 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.375976    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-model-cache\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.376293 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.376128    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-home\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.376436 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.376412    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.376585 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.376494    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.376684 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.376661    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.376747 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.376697    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-model-cache\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.378349 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.378324    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-dshm\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.378639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.378623    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tls-certs\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.387596 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.387574    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrfs\" (UniqueName: \"kubernetes.io/projected/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kube-api-access-qwrfs\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-h7pj4\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.535445 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.535400    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"]
Sep 16 19:14:51.538019 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.537996    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.555234 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.555201    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"]
Sep 16 19:14:51.578180 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.578142    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.578347 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.578193    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.578347 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.578259    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f60b6e18-caea-4ec8-813f-2001b02c2b55-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.578347 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.578308    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f60b6e18-caea-4ec8-813f-2001b02c2b55-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.582178 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.582151    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:51.596116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.595955    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:51.680050 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.680001    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f60b6e18-caea-4ec8-813f-2001b02c2b55-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.680228 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.680096    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f60b6e18-caea-4ec8-813f-2001b02c2b55-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.680228 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.680156    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.680228 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.680209    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.680900 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.680557    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.680900 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.680596    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.680900 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.680829    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f60b6e18-caea-4ec8-813f-2001b02c2b55-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.683142 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.683119    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f60b6e18-caea-4ec8-813f-2001b02c2b55-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.697616 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.697582    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:51.848058 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.847970    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:51.869082 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:51.869055    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"]
Sep 16 19:14:51.871995 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:14:51.871969    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e47ab7_ee8c_41af_8545_94e9dbc7f4cb.slice/crio-775ddf1e3e8c8bfa613ac128ee5c6a7e8e7bb92d999348006b2a97f5160375c4 WatchSource:0}: Error finding container 775ddf1e3e8c8bfa613ac128ee5c6a7e8e7bb92d999348006b2a97f5160375c4: Status 404 returned error can't find the container with id 775ddf1e3e8c8bfa613ac128ee5c6a7e8e7bb92d999348006b2a97f5160375c4
Sep 16 19:14:52.137001 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:52.136907    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"]
Sep 16 19:14:52.139853 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:14:52.139824    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60b6e18_caea_4ec8_813f_2001b02c2b55.slice/crio-c69e2ebb265f7d973973ef389c68c25668cca8b6e93dc752575e47df911644bb WatchSource:0}: Error finding container c69e2ebb265f7d973973ef389c68c25668cca8b6e93dc752575e47df911644bb: Status 404 returned error can't find the container with id c69e2ebb265f7d973973ef389c68c25668cca8b6e93dc752575e47df911644bb
Sep 16 19:14:52.429502 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:52.429405    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5" event={"ID":"f60b6e18-caea-4ec8-813f-2001b02c2b55","Type":"ContainerStarted","Data":"3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174"}
Sep 16 19:14:52.429502 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:52.429468    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5" event={"ID":"f60b6e18-caea-4ec8-813f-2001b02c2b55","Type":"ContainerStarted","Data":"c69e2ebb265f7d973973ef389c68c25668cca8b6e93dc752575e47df911644bb"}
Sep 16 19:14:52.431229 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:52.431197    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" event={"ID":"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb","Type":"ContainerStarted","Data":"f990890cf629ad7d3d0bb9817468e24cab85800eb3194c85f6b9466ae99340ff"}
Sep 16 19:14:52.431355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:52.431255    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" event={"ID":"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb","Type":"ContainerStarted","Data":"775ddf1e3e8c8bfa613ac128ee5c6a7e8e7bb92d999348006b2a97f5160375c4"}
Sep 16 19:14:55.449142 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:55.449107    2590 generic.go:356] "Generic (PLEG): container finished" podID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerID="f990890cf629ad7d3d0bb9817468e24cab85800eb3194c85f6b9466ae99340ff" exitCode=0
Sep 16 19:14:55.449580 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:55.449180    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" event={"ID":"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb","Type":"ContainerDied","Data":"f990890cf629ad7d3d0bb9817468e24cab85800eb3194c85f6b9466ae99340ff"}
Sep 16 19:14:55.450848 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:55.450827    2590 generic.go:356] "Generic (PLEG): container finished" podID="f60b6e18-caea-4ec8-813f-2001b02c2b55" containerID="3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174" exitCode=0
Sep 16 19:14:55.450951 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:55.450896    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5" event={"ID":"f60b6e18-caea-4ec8-813f-2001b02c2b55","Type":"ContainerDied","Data":"3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174"}
Sep 16 19:14:56.459631 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:56.459585    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5" event={"ID":"f60b6e18-caea-4ec8-813f-2001b02c2b55","Type":"ContainerStarted","Data":"ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0"}
Sep 16 19:14:56.460137 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:56.459846    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:14:56.461471 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:56.461448    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" event={"ID":"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb","Type":"ContainerStarted","Data":"8419c9e5e904877a61cf21fc8fb18f05ebb9c63d9f4e326773b72ae4709ec0c3"}
Sep 16 19:14:56.483671 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:56.483613    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5" podStartSLOduration=2.8321511299999997 podStartE2EDuration="5.48359476s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.65144363s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:56.483259463 +0000 UTC m=+2112.007718311" watchObservedRunningTime="2026-09-16 19:14:56.48359476 +0000 UTC m=+2112.008053596"
Sep 16 19:14:56.511443 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:56.511390    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" podStartSLOduration=2.8975403650000002 podStartE2EDuration="5.511372433s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.613832068s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:56.508402243 +0000 UTC m=+2112.032861080" watchObservedRunningTime="2026-09-16 19:14:56.511372433 +0000 UTC m=+2112.035831267"
Sep 16 19:14:56.582918 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:56.582865    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:14:57.638333 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:57.638298    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:57.780119 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:57.780072    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:57.889141 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:14:57.889051    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:01.582459 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:01.582414    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:15:01.595764 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:01.595732    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:15:02.503642 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:02.503613    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:15:02.691013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:02.690978    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:10.179535 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:10.179500    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:11.762335 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:11.762298    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:12.516407 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:12.516376    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:22.128534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:22.128497    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:23.351776 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:23.351721    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:27.477805 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:27.477766    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:15:37.135131 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.135037    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5_f60b6e18-caea-4ec8-813f-2001b02c2b55/main/0.log"
Sep 16 19:15:37.432658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.432564    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"]
Sep 16 19:15:37.432835 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.432802    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5" podUID="f60b6e18-caea-4ec8-813f-2001b02c2b55" containerName="main" containerID="cri-o://ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0" gracePeriod=30
Sep 16 19:15:37.441745 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.441718    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"]
Sep 16 19:15:37.450626 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:15:37.450588    2590 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:15:37.450626 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:37.450626 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:37.450626 ip-10-0-143-44 kubenswrapper[2590]:  > containerID="8419c9e5e904877a61cf21fc8fb18f05ebb9c63d9f4e326773b72ae4709ec0c3" cmd=["/bin/sleep","15"]
Sep 16 19:15:37.450863 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.450648    2590 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" message="" err=<
Sep 16 19:15:37.450863 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:37.450863 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:37.450863 ip-10-0-143-44 kubenswrapper[2590]:  >
Sep 16 19:15:37.450863 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:15:37.450665    2590 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:15:37.450863 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:37.450863 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:37.450863 ip-10-0-143-44 kubenswrapper[2590]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" podUID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerName="main" containerID="cri-o://8419c9e5e904877a61cf21fc8fb18f05ebb9c63d9f4e326773b72ae4709ec0c3"
Sep 16 19:15:37.450863 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.450723    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" podUID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerName="main" containerID="cri-o://8419c9e5e904877a61cf21fc8fb18f05ebb9c63d9f4e326773b72ae4709ec0c3" gracePeriod=60
Sep 16 19:15:37.465353 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.465322    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:37.492455 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.492422    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" podUID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:15:37.564761 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.564733    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:37.654077 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.654042    2590 generic.go:356] "Generic (PLEG): container finished" podID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerID="8419c9e5e904877a61cf21fc8fb18f05ebb9c63d9f4e326773b72ae4709ec0c3" exitCode=0
Sep 16 19:15:37.654242 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:37.654116    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" event={"ID":"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb","Type":"ContainerDied","Data":"8419c9e5e904877a61cf21fc8fb18f05ebb9c63d9f4e326773b72ae4709ec0c3"}
Sep 16 19:15:38.080451 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.080423    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:15:38.103650 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.103209    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f60b6e18-caea-4ec8-813f-2001b02c2b55-cabundle-cert\") pod \"f60b6e18-caea-4ec8-813f-2001b02c2b55\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") "
Sep 16 19:15:38.103650 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.103266    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-tmp-dir\") pod \"f60b6e18-caea-4ec8-813f-2001b02c2b55\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") "
Sep 16 19:15:38.103650 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.103323    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-kserve-provision-location\") pod \"f60b6e18-caea-4ec8-813f-2001b02c2b55\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") "
Sep 16 19:15:38.103650 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.103354    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f60b6e18-caea-4ec8-813f-2001b02c2b55-tls-certs\") pod \"f60b6e18-caea-4ec8-813f-2001b02c2b55\" (UID: \"f60b6e18-caea-4ec8-813f-2001b02c2b55\") "
Sep 16 19:15:38.104122 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.103870    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60b6e18-caea-4ec8-813f-2001b02c2b55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f60b6e18-caea-4ec8-813f-2001b02c2b55" (UID: "f60b6e18-caea-4ec8-813f-2001b02c2b55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:38.104275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.104226    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f60b6e18-caea-4ec8-813f-2001b02c2b55" (UID: "f60b6e18-caea-4ec8-813f-2001b02c2b55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:38.107827 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.107748    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60b6e18-caea-4ec8-813f-2001b02c2b55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f60b6e18-caea-4ec8-813f-2001b02c2b55" (UID: "f60b6e18-caea-4ec8-813f-2001b02c2b55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:38.128979 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.128931    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f60b6e18-caea-4ec8-813f-2001b02c2b55" (UID: "f60b6e18-caea-4ec8-813f-2001b02c2b55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:38.204573 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.204489    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f60b6e18-caea-4ec8-813f-2001b02c2b55-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.204573 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.204516    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.204573 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.204527    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f60b6e18-caea-4ec8-813f-2001b02c2b55-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.204573 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.204536    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f60b6e18-caea-4ec8-813f-2001b02c2b55-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.215673 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.215642    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:15:38.305413 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305378    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-dshm\") pod \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") "
Sep 16 19:15:38.305587 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305439    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kserve-provision-location\") pod \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") "
Sep 16 19:15:38.305587 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305476    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-home\") pod \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") "
Sep 16 19:15:38.305587 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305501    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tmp-dir\") pod \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") "
Sep 16 19:15:38.305587 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305555    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-cabundle-cert\") pod \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") "
Sep 16 19:15:38.305800 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305591    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tls-certs\") pod \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") "
Sep 16 19:15:38.305800 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305643    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-model-cache\") pod \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") "
Sep 16 19:15:38.305800 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305680    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwrfs\" (UniqueName: \"kubernetes.io/projected/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kube-api-access-qwrfs\") pod \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\" (UID: \"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb\") "
Sep 16 19:15:38.306004 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305815    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" (UID: "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:38.306004 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.305835    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-home" (OuterVolumeSpecName: "home") pod "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" (UID: "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:38.306138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.306018    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" (UID: "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:38.306138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.306014    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.306138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.306044    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" (UID: "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:38.306138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.306055    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.307677 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.307648    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-dshm" (OuterVolumeSpecName: "dshm") pod "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" (UID: "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:38.307914 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.307871    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" (UID: "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:38.308216 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.308189    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kube-api-access-qwrfs" (OuterVolumeSpecName: "kube-api-access-qwrfs") pod "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" (UID: "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb"). InnerVolumeSpecName "kube-api-access-qwrfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:38.329787 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.329742    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" (UID: "13e47ab7-ee8c-41af-8545-94e9dbc7f4cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:38.406794 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.406756    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.406794 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.406789    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.406794 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.406800    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.407057 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.406809    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.407057 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.406817    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.407057 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.406824    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrfs\" (UniqueName: \"kubernetes.io/projected/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb-kube-api-access-qwrfs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.665114 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.665072    2590 generic.go:356] "Generic (PLEG): container finished" podID="f60b6e18-caea-4ec8-813f-2001b02c2b55" containerID="ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0" exitCode=0
Sep 16 19:15:38.665297 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.665193    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"
Sep 16 19:15:38.665297 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.665211    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5" event={"ID":"f60b6e18-caea-4ec8-813f-2001b02c2b55","Type":"ContainerDied","Data":"ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0"}
Sep 16 19:15:38.665397 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.665306    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5" event={"ID":"f60b6e18-caea-4ec8-813f-2001b02c2b55","Type":"ContainerDied","Data":"c69e2ebb265f7d973973ef389c68c25668cca8b6e93dc752575e47df911644bb"}
Sep 16 19:15:38.665397 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.665339    2590 scope.go:117] "RemoveContainer" containerID="ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0"
Sep 16 19:15:38.667079 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.667038    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4" event={"ID":"13e47ab7-ee8c-41af-8545-94e9dbc7f4cb","Type":"ContainerDied","Data":"775ddf1e3e8c8bfa613ac128ee5c6a7e8e7bb92d999348006b2a97f5160375c4"}
Sep 16 19:15:38.667185 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.667080    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"
Sep 16 19:15:38.674847 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.674832    2590 scope.go:117] "RemoveContainer" containerID="3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174"
Sep 16 19:15:38.682625 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.682608    2590 scope.go:117] "RemoveContainer" containerID="ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0"
Sep 16 19:15:38.682869 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:15:38.682849    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0\": container with ID starting with ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0 not found: ID does not exist" containerID="ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0"
Sep 16 19:15:38.682934 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.682895    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0"} err="failed to get container status \"ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0\": rpc error: code = NotFound desc = could not find container \"ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0\": container with ID starting with ac0d4701abdda5f0784e9be707d3d65312d57a2f48f7864aa8dc45396b4e03d0 not found: ID does not exist"
Sep 16 19:15:38.682934 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.682914    2590 scope.go:117] "RemoveContainer" containerID="3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174"
Sep 16 19:15:38.683174 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:15:38.683154    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174\": container with ID starting with 3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174 not found: ID does not exist" containerID="3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174"
Sep 16 19:15:38.683255 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.683184    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174"} err="failed to get container status \"3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174\": rpc error: code = NotFound desc = could not find container \"3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174\": container with ID starting with 3dd69045cd2a110bcd2f536aff269fc5086a9738320b4fb5c659a7ba697bd174 not found: ID does not exist"
Sep 16 19:15:38.683255 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.683208    2590 scope.go:117] "RemoveContainer" containerID="8419c9e5e904877a61cf21fc8fb18f05ebb9c63d9f4e326773b72ae4709ec0c3"
Sep 16 19:15:38.691302 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.691282    2590 scope.go:117] "RemoveContainer" containerID="f990890cf629ad7d3d0bb9817468e24cab85800eb3194c85f6b9466ae99340ff"
Sep 16 19:15:38.693116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.693098    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"]
Sep 16 19:15:38.700408 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.700385    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84c74b66bd-g2vs5"]
Sep 16 19:15:38.715707 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.715685    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"]
Sep 16 19:15:38.721433 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:38.721410    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-h7pj4"]
Sep 16 19:15:39.025671 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:39.025637    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" path="/var/lib/kubelet/pods/13e47ab7-ee8c-41af-8545-94e9dbc7f4cb/volumes"
Sep 16 19:15:39.026117 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:39.026102    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f60b6e18-caea-4ec8-813f-2001b02c2b55" path="/var/lib/kubelet/pods/f60b6e18-caea-4ec8-813f-2001b02c2b55/volumes"
Sep 16 19:15:40.928103 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:40.928063    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:46.980253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:46.980218    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:47.216601 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:47.216561    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:50.660720 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.660678    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"]
Sep 16 19:15:50.661208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661091    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerName="storage-initializer"
Sep 16 19:15:50.661208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661110    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerName="storage-initializer"
Sep 16 19:15:50.661208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661122    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60b6e18-caea-4ec8-813f-2001b02c2b55" containerName="storage-initializer"
Sep 16 19:15:50.661208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661130    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60b6e18-caea-4ec8-813f-2001b02c2b55" containerName="storage-initializer"
Sep 16 19:15:50.661208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661155    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60b6e18-caea-4ec8-813f-2001b02c2b55" containerName="main"
Sep 16 19:15:50.661208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661165    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60b6e18-caea-4ec8-813f-2001b02c2b55" containerName="main"
Sep 16 19:15:50.661208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661177    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerName="main"
Sep 16 19:15:50.661208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661185    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerName="main"
Sep 16 19:15:50.661597 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661260    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60b6e18-caea-4ec8-813f-2001b02c2b55" containerName="main"
Sep 16 19:15:50.661597 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.661276    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e47ab7-ee8c-41af-8545-94e9dbc7f4cb" containerName="main"
Sep 16 19:15:50.664223 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.664202    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.666954 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.666936    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:15:50.667074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.666938    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:15:50.667627 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.667603    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-bwc2w\""
Sep 16 19:15:50.667715 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.667640    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 16 19:15:50.684962 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.684936    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"]
Sep 16 19:15:50.712743 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.712699    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchpc\" (UniqueName: \"kubernetes.io/projected/9ac5c311-ba1a-4298-832e-64f02dfe576f-kube-api-access-dchpc\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.712929 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.712814    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-tmp-dir\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.712929 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.712913    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-home\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.713045 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.712963    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-model-cache\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.713045 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.712988    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-dshm\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.713045 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.713026    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac5c311-ba1a-4298-832e-64f02dfe576f-tls-certs\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.724609 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.724588    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:50.813459 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.813423    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-tmp-dir\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.813648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.813482    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-home\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.813648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.813516    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-model-cache\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.813648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.813534    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-dshm\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.813648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.813559    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac5c311-ba1a-4298-832e-64f02dfe576f-tls-certs\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.813648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.813594    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchpc\" (UniqueName: \"kubernetes.io/projected/9ac5c311-ba1a-4298-832e-64f02dfe576f-kube-api-access-dchpc\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.813973 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.813951    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-tmp-dir\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.814039 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.813963    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-home\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.814039 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.814023    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-model-cache\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.816018 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.815996    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-dshm\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.816153 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.816132    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac5c311-ba1a-4298-832e-64f02dfe576f-tls-certs\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.824497 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.824475    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchpc\" (UniqueName: \"kubernetes.io/projected/9ac5c311-ba1a-4298-832e-64f02dfe576f-kube-api-access-dchpc\") pod \"late-v2-kserve-86766546f-kpw5q\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:50.975264 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:50.975160    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:15:51.255488 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:51.255451    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"]
Sep 16 19:15:51.257964 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:15:51.257932    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac5c311_ba1a_4298_832e_64f02dfe576f.slice/crio-cdb0587db4db6a2601cbab61b059d837f48a8eebc54eeee3cde6a89880ba6596 WatchSource:0}: Error finding container cdb0587db4db6a2601cbab61b059d837f48a8eebc54eeee3cde6a89880ba6596: Status 404 returned error can't find the container with id cdb0587db4db6a2601cbab61b059d837f48a8eebc54eeee3cde6a89880ba6596
Sep 16 19:15:51.726251 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:51.726208    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q" event={"ID":"9ac5c311-ba1a-4298-832e-64f02dfe576f","Type":"ContainerStarted","Data":"cdb0587db4db6a2601cbab61b059d837f48a8eebc54eeee3cde6a89880ba6596"}
Sep 16 19:15:52.732598 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:52.732503    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q" event={"ID":"9ac5c311-ba1a-4298-832e-64f02dfe576f","Type":"ContainerStarted","Data":"3e6d92dc0bdc08a256c18c64633dabd353d0b6d0a77c50a24d97f2c5dae898cf"}
Sep 16 19:15:52.751818 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:52.751766    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q" podStartSLOduration=1.600508423 podStartE2EDuration="2.751747835s" totalImagesPullingTime="1.151239412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:52.75053005 +0000 UTC m=+2168.274988883" watchObservedRunningTime="2026-09-16 19:15:52.751747835 +0000 UTC m=+2168.276206689"
Sep 16 19:15:52.976353 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:15:52.976310    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:16:00.976244 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:00.976201    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:16:00.988851 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:00.988821    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:16:01.781354 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:01.781320    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:16:04.160013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:04.159983    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:27.460296 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:27.460258    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:28.737588 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:28.737553    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:33.852459 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:33.852409    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:33.959747 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:33.959704    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:34.054051 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:34.054018    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:35.573629 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:35.573593    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:37.782261 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.782227    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"]
Sep 16 19:16:37.785608 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.785587    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.787844 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.787823    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:16:37.788487 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.788469    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:16:37.789358 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.789340    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:16:37.789802 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.789766    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:16:37.789910 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.789766    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:16:37.789910 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.789818    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-kdv7p\""
Sep 16 19:16:37.807522 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.803244    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"]
Sep 16 19:16:37.812654 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.812621    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:37.885417 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.885378    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tls-certs\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.885417 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.885426    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-model-cache\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.885738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.885445    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.885738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.885500    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kube-api-access-t579x\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.885738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.885526    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-home\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.885738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.885543    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tmp-dir\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.885738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.885599    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.885738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.885633    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-dshm\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986156 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986110    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tls-certs\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986156 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986164    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-model-cache\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986419 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986186    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986419 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986222    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kube-api-access-t579x\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986419 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986262    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-home\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986419 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986287    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tmp-dir\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986419 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986322    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986419 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986347    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-dshm\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986749 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986621    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-model-cache\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986749 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986666    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-home\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986828 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986744    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tmp-dir\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.986915 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.986872    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.987088 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.987073    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.988668 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.988649    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-dshm\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.988984 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.988965    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tls-certs\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:37.995306 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:37.995281    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kube-api-access-t579x\") pod \"tokenizer-template-args-test-kserve-67c45f5458-v5jrf\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:38.065549 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.065471    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"]
Sep 16 19:16:38.069164 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.069147    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.084937 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.084906    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"]
Sep 16 19:16:38.087510 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.087486    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.087638 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.087518    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e48885-4e80-47e8-9f7c-676ab581030f-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.087638 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.087553    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.087741 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.087647    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e48885-4e80-47e8-9f7c-676ab581030f-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.095490 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.095295    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:38.138926 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.138860    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:38.188519 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.188482    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.188519 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.188522    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e48885-4e80-47e8-9f7c-676ab581030f-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.188780 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.188554    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.188780 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.188589    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e48885-4e80-47e8-9f7c-676ab581030f-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.189048 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.189021    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.189124 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.189051    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.189276 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.189259    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e48885-4e80-47e8-9f7c-676ab581030f-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.191283 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.191264    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e48885-4e80-47e8-9f7c-676ab581030f-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-757d687b88-8p5zk\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.227761 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.227726    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:38.371796 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.371770    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"]
Sep 16 19:16:38.374108 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:16:38.374078    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode034297f_e63f_4a4c_8b6a_0ddc7eb346d8.slice/crio-2e6346cc8215ac78ddcc6d1025a9c7036aee73897a5276de0cc510b5c2bf6e0d WatchSource:0}: Error finding container 2e6346cc8215ac78ddcc6d1025a9c7036aee73897a5276de0cc510b5c2bf6e0d: Status 404 returned error can't find the container with id 2e6346cc8215ac78ddcc6d1025a9c7036aee73897a5276de0cc510b5c2bf6e0d
Sep 16 19:16:38.379980 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.379957    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:38.659442 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.659416    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"]
Sep 16 19:16:38.662342 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:16:38.662314    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e48885_4e80_47e8_9f7c_676ab581030f.slice/crio-0a45693d4931f28f408c0e6838563f06034387a67dfb6770b77a30bc8dba153e WatchSource:0}: Error finding container 0a45693d4931f28f408c0e6838563f06034387a67dfb6770b77a30bc8dba153e: Status 404 returned error can't find the container with id 0a45693d4931f28f408c0e6838563f06034387a67dfb6770b77a30bc8dba153e
Sep 16 19:16:38.841584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.841545    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"]
Sep 16 19:16:38.938958 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.938602    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk" event={"ID":"d4e48885-4e80-47e8-9f7c-676ab581030f","Type":"ContainerStarted","Data":"9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6"}
Sep 16 19:16:38.938958 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.938680    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk" event={"ID":"d4e48885-4e80-47e8-9f7c-676ab581030f","Type":"ContainerStarted","Data":"0a45693d4931f28f408c0e6838563f06034387a67dfb6770b77a30bc8dba153e"}
Sep 16 19:16:38.940034 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.940000    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:38.940521 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.940497    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" event={"ID":"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8","Type":"ContainerStarted","Data":"84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775"}
Sep 16 19:16:38.940635 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:38.940552    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" event={"ID":"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8","Type":"ContainerStarted","Data":"2e6346cc8215ac78ddcc6d1025a9c7036aee73897a5276de0cc510b5c2bf6e0d"}
Sep 16 19:16:40.363170 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:40.363127    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:41.449626 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:41.449587    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:41.960157 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:41.960124    2590 generic.go:356] "Generic (PLEG): container finished" podID="d4e48885-4e80-47e8-9f7c-676ab581030f" containerID="9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6" exitCode=0
Sep 16 19:16:41.960343 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:41.960197    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk" event={"ID":"d4e48885-4e80-47e8-9f7c-676ab581030f","Type":"ContainerDied","Data":"9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6"}
Sep 16 19:16:41.961870 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:41.961842    2590 generic.go:356] "Generic (PLEG): container finished" podID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" containerID="84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775" exitCode=0
Sep 16 19:16:41.962021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:41.961897    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" event={"ID":"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8","Type":"ContainerDied","Data":"84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775"}
Sep 16 19:16:42.540629 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:42.540598    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:42.640462 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:42.640428    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:42.969012 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:42.968898    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" event={"ID":"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8","Type":"ContainerStarted","Data":"2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f"}
Sep 16 19:16:42.970996 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:42.970971    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk" event={"ID":"d4e48885-4e80-47e8-9f7c-676ab581030f","Type":"ContainerStarted","Data":"800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3"}
Sep 16 19:16:42.971187 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:42.971171    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:16:43.037381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:43.037329    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" podStartSLOduration=3.448577243 podStartE2EDuration="6.037313054s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.588735811s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:43.035709885 +0000 UTC m=+2218.560168719" watchObservedRunningTime="2026-09-16 19:16:43.037313054 +0000 UTC m=+2218.561771887"
Sep 16 19:16:43.073208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:43.073159    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk" podStartSLOduration=2.480554446 podStartE2EDuration="5.073144758s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.592590312s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:43.072427511 +0000 UTC m=+2218.596886346" watchObservedRunningTime="2026-09-16 19:16:43.073144758 +0000 UTC m=+2218.597603592"
Sep 16 19:16:43.095476 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:43.095440    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:44.838714 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:44.838670    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:44.938711 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:44.938677    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:48.096496 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:48.096460    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:48.109159 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:48.109129    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:49.010329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:49.010294    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:16:50.495377 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:50.495343    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:53.861308 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:53.861262    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q" podUID="9ac5c311-ba1a-4298-832e-64f02dfe576f" containerName="main" containerID="cri-o://3e6d92dc0bdc08a256c18c64633dabd353d0b6d0a77c50a24d97f2c5dae898cf" gracePeriod=45
Sep 16 19:16:54.031335 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.031294    2590 generic.go:356] "Generic (PLEG): container finished" podID="9ac5c311-ba1a-4298-832e-64f02dfe576f" containerID="3e6d92dc0bdc08a256c18c64633dabd353d0b6d0a77c50a24d97f2c5dae898cf" exitCode=0
Sep 16 19:16:54.031499 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.031356    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q" event={"ID":"9ac5c311-ba1a-4298-832e-64f02dfe576f","Type":"ContainerDied","Data":"3e6d92dc0bdc08a256c18c64633dabd353d0b6d0a77c50a24d97f2c5dae898cf"}
Sep 16 19:16:54.110310 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.110282    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:16:54.228636 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.228542    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-dshm\") pod \"9ac5c311-ba1a-4298-832e-64f02dfe576f\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") "
Sep 16 19:16:54.228636 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.228606    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac5c311-ba1a-4298-832e-64f02dfe576f-tls-certs\") pod \"9ac5c311-ba1a-4298-832e-64f02dfe576f\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") "
Sep 16 19:16:54.228875 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.228649    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-tmp-dir\") pod \"9ac5c311-ba1a-4298-832e-64f02dfe576f\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") "
Sep 16 19:16:54.228875 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.228676    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dchpc\" (UniqueName: \"kubernetes.io/projected/9ac5c311-ba1a-4298-832e-64f02dfe576f-kube-api-access-dchpc\") pod \"9ac5c311-ba1a-4298-832e-64f02dfe576f\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") "
Sep 16 19:16:54.228875 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.228694    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-home\") pod \"9ac5c311-ba1a-4298-832e-64f02dfe576f\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") "
Sep 16 19:16:54.228875 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.228723    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-model-cache\") pod \"9ac5c311-ba1a-4298-832e-64f02dfe576f\" (UID: \"9ac5c311-ba1a-4298-832e-64f02dfe576f\") "
Sep 16 19:16:54.229106 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.229022    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-home" (OuterVolumeSpecName: "home") pod "9ac5c311-ba1a-4298-832e-64f02dfe576f" (UID: "9ac5c311-ba1a-4298-832e-64f02dfe576f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:54.229106 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.229039    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ac5c311-ba1a-4298-832e-64f02dfe576f" (UID: "9ac5c311-ba1a-4298-832e-64f02dfe576f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:54.229216 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.229165    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ac5c311-ba1a-4298-832e-64f02dfe576f" (UID: "9ac5c311-ba1a-4298-832e-64f02dfe576f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:54.231131 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.231104    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-dshm" (OuterVolumeSpecName: "dshm") pod "9ac5c311-ba1a-4298-832e-64f02dfe576f" (UID: "9ac5c311-ba1a-4298-832e-64f02dfe576f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:54.231256 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.231163    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac5c311-ba1a-4298-832e-64f02dfe576f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ac5c311-ba1a-4298-832e-64f02dfe576f" (UID: "9ac5c311-ba1a-4298-832e-64f02dfe576f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:54.231329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.231312    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac5c311-ba1a-4298-832e-64f02dfe576f-kube-api-access-dchpc" (OuterVolumeSpecName: "kube-api-access-dchpc") pod "9ac5c311-ba1a-4298-832e-64f02dfe576f" (UID: "9ac5c311-ba1a-4298-832e-64f02dfe576f"). InnerVolumeSpecName "kube-api-access-dchpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:54.329921 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.329874    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.329921 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.329914    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.330133 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.329943    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac5c311-ba1a-4298-832e-64f02dfe576f-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.330133 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.329952    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.330133 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.329961    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchpc\" (UniqueName: \"kubernetes.io/projected/9ac5c311-ba1a-4298-832e-64f02dfe576f-kube-api-access-dchpc\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.330133 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:54.329969    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac5c311-ba1a-4298-832e-64f02dfe576f-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:55.037325 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:55.037304    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"
Sep 16 19:16:55.037660 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:55.037299    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q" event={"ID":"9ac5c311-ba1a-4298-832e-64f02dfe576f","Type":"ContainerDied","Data":"cdb0587db4db6a2601cbab61b059d837f48a8eebc54eeee3cde6a89880ba6596"}
Sep 16 19:16:55.037660 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:55.037454    2590 scope.go:117] "RemoveContainer" containerID="3e6d92dc0bdc08a256c18c64633dabd353d0b6d0a77c50a24d97f2c5dae898cf"
Sep 16 19:16:55.054694 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:55.054666    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"]
Sep 16 19:16:55.060369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:55.060343    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kpw5q"]
Sep 16 19:16:57.025683 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:16:57.025639    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ac5c311-ba1a-4298-832e-64f02dfe576f" path="/var/lib/kubelet/pods/9ac5c311-ba1a-4298-832e-64f02dfe576f/volumes"
Sep 16 19:17:13.986214 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:13.986124    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:17:15.421453 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:15.421423    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:17.386403 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:17.386368    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:20.311143 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:20.311106    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:21.985208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:21.985178    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-757d687b88-8p5zk_d4e48885-4e80-47e8-9f7c-676ab581030f/main/0.log"
Sep 16 19:17:24.604192 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:24.604142    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:24.680298 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:24.680252    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:41.521265 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:41.521229    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:41.623049 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:41.623006    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:51.199105 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.199072    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:51.245711 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.245680    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"]
Sep 16 19:17:51.245962 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.245940    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk" podUID="d4e48885-4e80-47e8-9f7c-676ab581030f" containerName="main" containerID="cri-o://800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3" gracePeriod=30
Sep 16 19:17:51.253534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.253507    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"]
Sep 16 19:17:51.262729 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:17:51.262694    2590 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:17:51.262729 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:51.262729 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:51.262729 ip-10-0-143-44 kubenswrapper[2590]:  > containerID="2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f" cmd=["/bin/sleep","15"]
Sep 16 19:17:51.262981 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.262746    2590 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" message="" err=<
Sep 16 19:17:51.262981 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:51.262981 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:51.262981 ip-10-0-143-44 kubenswrapper[2590]:  >
Sep 16 19:17:51.262981 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:17:51.262758    2590 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:17:51.262981 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:51.262981 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:51.262981 ip-10-0-143-44 kubenswrapper[2590]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" podUID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" containerName="main" containerID="cri-o://2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f"
Sep 16 19:17:51.262981 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.262802    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" podUID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" containerName="main" containerID="cri-o://2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f" gracePeriod=60
Sep 16 19:17:51.282656 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.282627    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:51.350534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.350501    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:51.609729 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.609705    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:17:51.737422 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737335    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tls-certs\") pod \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") "
Sep 16 19:17:51.737422 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737395    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-dshm\") pod \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") "
Sep 16 19:17:51.737422 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737420    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-home\") pod \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") "
Sep 16 19:17:51.737707 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737451    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-model-cache\") pod \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") "
Sep 16 19:17:51.737707 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737467    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-cabundle-cert\") pod \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") "
Sep 16 19:17:51.737707 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737489    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kserve-provision-location\") pod \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") "
Sep 16 19:17:51.737707 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737529    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kube-api-access-t579x\") pod \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") "
Sep 16 19:17:51.737707 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737558    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tmp-dir\") pod \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\" (UID: \"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8\") "
Sep 16 19:17:51.738010 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737708    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" (UID: "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.738010 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737722    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-home" (OuterVolumeSpecName: "home") pod "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" (UID: "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.738010 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737767    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" (UID: "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:51.738010 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737873    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.738010 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737908    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.738010 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737912    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" (UID: "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.738010 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.737921    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.743075 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.743037    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-dshm" (OuterVolumeSpecName: "dshm") pod "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" (UID: "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.743223 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.743145    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kube-api-access-t579x" (OuterVolumeSpecName: "kube-api-access-t579x") pod "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" (UID: "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8"). InnerVolumeSpecName "kube-api-access-t579x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:51.744531 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.744502    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" (UID: "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:51.760589 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.760552    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" (UID: "e034297f-e63f-4a4c-8b6a-0ddc7eb346d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.841619 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.839809    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.841619 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.839859    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.841619 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.839906    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-kube-api-access-t579x\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.841619 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.839927    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.841619 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.839944    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.901269 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:51.901242    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:17:52.041954 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.041910    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e48885-4e80-47e8-9f7c-676ab581030f-tls-certs\") pod \"d4e48885-4e80-47e8-9f7c-676ab581030f\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") "
Sep 16 19:17:52.042159 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.041971    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-kserve-provision-location\") pod \"d4e48885-4e80-47e8-9f7c-676ab581030f\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") "
Sep 16 19:17:52.042159 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.041998    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e48885-4e80-47e8-9f7c-676ab581030f-cabundle-cert\") pod \"d4e48885-4e80-47e8-9f7c-676ab581030f\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") "
Sep 16 19:17:52.042159 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.042040    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-tmp-dir\") pod \"d4e48885-4e80-47e8-9f7c-676ab581030f\" (UID: \"d4e48885-4e80-47e8-9f7c-676ab581030f\") "
Sep 16 19:17:52.042437 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.042412    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e48885-4e80-47e8-9f7c-676ab581030f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4e48885-4e80-47e8-9f7c-676ab581030f" (UID: "d4e48885-4e80-47e8-9f7c-676ab581030f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:52.042437 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.042425    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4e48885-4e80-47e8-9f7c-676ab581030f" (UID: "d4e48885-4e80-47e8-9f7c-676ab581030f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:52.044299 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.044274    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e48885-4e80-47e8-9f7c-676ab581030f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4e48885-4e80-47e8-9f7c-676ab581030f" (UID: "d4e48885-4e80-47e8-9f7c-676ab581030f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:52.064599 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.064561    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4e48885-4e80-47e8-9f7c-676ab581030f" (UID: "d4e48885-4e80-47e8-9f7c-676ab581030f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:52.143099 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.143053    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e48885-4e80-47e8-9f7c-676ab581030f-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:52.143099 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.143096    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:52.143099 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.143106    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e48885-4e80-47e8-9f7c-676ab581030f-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:52.143099 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.143115    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e48885-4e80-47e8-9f7c-676ab581030f-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:52.290592 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.290556    2590 generic.go:356] "Generic (PLEG): container finished" podID="d4e48885-4e80-47e8-9f7c-676ab581030f" containerID="800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3" exitCode=0
Sep 16 19:17:52.291096 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.290615    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"
Sep 16 19:17:52.291096 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.290611    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk" event={"ID":"d4e48885-4e80-47e8-9f7c-676ab581030f","Type":"ContainerDied","Data":"800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3"}
Sep 16 19:17:52.291096 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.290747    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk" event={"ID":"d4e48885-4e80-47e8-9f7c-676ab581030f","Type":"ContainerDied","Data":"0a45693d4931f28f408c0e6838563f06034387a67dfb6770b77a30bc8dba153e"}
Sep 16 19:17:52.291096 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.290783    2590 scope.go:117] "RemoveContainer" containerID="800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3"
Sep 16 19:17:52.292527 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.292464    2590 generic.go:356] "Generic (PLEG): container finished" podID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" containerID="2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f" exitCode=0
Sep 16 19:17:52.292620 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.292542    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" event={"ID":"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8","Type":"ContainerDied","Data":"2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f"}
Sep 16 19:17:52.292620 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.292581    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf" event={"ID":"e034297f-e63f-4a4c-8b6a-0ddc7eb346d8","Type":"ContainerDied","Data":"2e6346cc8215ac78ddcc6d1025a9c7036aee73897a5276de0cc510b5c2bf6e0d"}
Sep 16 19:17:52.292620 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.292546    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"
Sep 16 19:17:52.299862 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.299650    2590 scope.go:117] "RemoveContainer" containerID="9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6"
Sep 16 19:17:52.307644 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.307625    2590 scope.go:117] "RemoveContainer" containerID="800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3"
Sep 16 19:17:52.307951 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:17:52.307925    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3\": container with ID starting with 800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3 not found: ID does not exist" containerID="800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3"
Sep 16 19:17:52.308001 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.307963    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3"} err="failed to get container status \"800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3\": rpc error: code = NotFound desc = could not find container \"800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3\": container with ID starting with 800937599a80d0fc89250a4bc91d1507fd4f8283959b70d99a4dfdd9bee5c2e3 not found: ID does not exist"
Sep 16 19:17:52.308001 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.307985    2590 scope.go:117] "RemoveContainer" containerID="9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6"
Sep 16 19:17:52.308259 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:17:52.308236    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6\": container with ID starting with 9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6 not found: ID does not exist" containerID="9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6"
Sep 16 19:17:52.308370 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.308263    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6"} err="failed to get container status \"9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6\": rpc error: code = NotFound desc = could not find container \"9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6\": container with ID starting with 9b72ed6228322170fb476692a8957ec0ab2585115fafe877040399222b78ead6 not found: ID does not exist"
Sep 16 19:17:52.308370 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.308285    2590 scope.go:117] "RemoveContainer" containerID="2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f"
Sep 16 19:17:52.316005 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.315989    2590 scope.go:117] "RemoveContainer" containerID="84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775"
Sep 16 19:17:52.323308 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.323291    2590 scope.go:117] "RemoveContainer" containerID="2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f"
Sep 16 19:17:52.323601 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:17:52.323576    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f\": container with ID starting with 2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f not found: ID does not exist" containerID="2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f"
Sep 16 19:17:52.323676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.323610    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f"} err="failed to get container status \"2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f\": rpc error: code = NotFound desc = could not find container \"2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f\": container with ID starting with 2706d9d3602cf843abfc6c48e0751830b67c3aa46a19b20f9fa939d579aa496f not found: ID does not exist"
Sep 16 19:17:52.323676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.323631    2590 scope.go:117] "RemoveContainer" containerID="84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775"
Sep 16 19:17:52.323946 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:17:52.323931    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775\": container with ID starting with 84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775 not found: ID does not exist" containerID="84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775"
Sep 16 19:17:52.324015 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.323951    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775"} err="failed to get container status \"84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775\": rpc error: code = NotFound desc = could not find container \"84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775\": container with ID starting with 84d83cf07a4ff70550309bfad39268b458bba59f37961798647a0486fa704775 not found: ID does not exist"
Sep 16 19:17:52.345204 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.345178    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"]
Sep 16 19:17:52.353549 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.353524    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-757d687b88-8p5zk"]
Sep 16 19:17:52.372040 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.372008    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"]
Sep 16 19:17:52.378017 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:52.377988    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-v5jrf"]
Sep 16 19:17:53.026181 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:53.026139    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e48885-4e80-47e8-9f7c-676ab581030f" path="/var/lib/kubelet/pods/d4e48885-4e80-47e8-9f7c-676ab581030f/volumes"
Sep 16 19:17:53.026679 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:17:53.026658    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" path="/var/lib/kubelet/pods/e034297f-e63f-4a4c-8b6a-0ddc7eb346d8/volumes"
Sep 16 19:18:03.880907 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:03.880845    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:05.787350 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:05.787309    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:07.976274 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:07.976240    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:08.075577 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:08.075538    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:21.966457 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:21.966411    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:22.105455 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:22.105414    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:22.209067 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:22.209027    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:26.023205 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023166    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"]
Sep 16 19:18:26.023766 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023745    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" containerName="storage-initializer"
Sep 16 19:18:26.023838 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023778    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" containerName="storage-initializer"
Sep 16 19:18:26.023838 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023793    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e48885-4e80-47e8-9f7c-676ab581030f" containerName="storage-initializer"
Sep 16 19:18:26.023838 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023801    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e48885-4e80-47e8-9f7c-676ab581030f" containerName="storage-initializer"
Sep 16 19:18:26.023838 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023837    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" containerName="main"
Sep 16 19:18:26.024067 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023847    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" containerName="main"
Sep 16 19:18:26.024067 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023864    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac5c311-ba1a-4298-832e-64f02dfe576f" containerName="main"
Sep 16 19:18:26.024067 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023872    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac5c311-ba1a-4298-832e-64f02dfe576f" containerName="main"
Sep 16 19:18:26.024067 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023981    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e48885-4e80-47e8-9f7c-676ab581030f" containerName="main"
Sep 16 19:18:26.024067 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.023998    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e48885-4e80-47e8-9f7c-676ab581030f" containerName="main"
Sep 16 19:18:26.024288 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.024156    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="e034297f-e63f-4a4c-8b6a-0ddc7eb346d8" containerName="main"
Sep 16 19:18:26.024288 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.024174    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e48885-4e80-47e8-9f7c-676ab581030f" containerName="main"
Sep 16 19:18:26.024288 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.024185    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac5c311-ba1a-4298-832e-64f02dfe576f" containerName="main"
Sep 16 19:18:26.029403 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.029376    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.037427 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.037404    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 16 19:18:26.038215 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.038190    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 19:18:26.038345 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.038221    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-lkgcb\""
Sep 16 19:18:26.038345 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.038190    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 19:18:26.064291 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.064256    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"]
Sep 16 19:18:26.118770 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.118729    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.119033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.118783    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.119033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.118856    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f10903e-be9d-4be6-a530-7aa76cd894e1-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.119033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.118943    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-dshm\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.119215 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.119113    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-home\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.119215 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.119164    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gjmq\" (UniqueName: \"kubernetes.io/projected/3f10903e-be9d-4be6-a530-7aa76cd894e1-kube-api-access-8gjmq\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.120148 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.120127    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:26.220287 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.220244    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f10903e-be9d-4be6-a530-7aa76cd894e1-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.220287 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.220289    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-dshm\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.220515 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.220412    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-home\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.220515 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.220460    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gjmq\" (UniqueName: \"kubernetes.io/projected/3f10903e-be9d-4be6-a530-7aa76cd894e1-kube-api-access-8gjmq\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.220515 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.220498    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.220638 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.220535    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.220789 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.220760    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-home\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.220949 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.220854    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.220949 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.220905    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.223273 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.223252    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-dshm\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.223517 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.223500    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f10903e-be9d-4be6-a530-7aa76cd894e1-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.241453 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.241421    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gjmq\" (UniqueName: \"kubernetes.io/projected/3f10903e-be9d-4be6-a530-7aa76cd894e1-kube-api-access-8gjmq\") pod \"roll-v1-kserve-69cf8bcfc-78qh8\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.340704 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.340665    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:26.619928 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:26.619903    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"]
Sep 16 19:18:26.621547 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:18:26.621519    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f10903e_be9d_4be6_a530_7aa76cd894e1.slice/crio-5c2bb9dca37002b38901ec7987ad2018206d62e70590e41556525591cc98c34a WatchSource:0}: Error finding container 5c2bb9dca37002b38901ec7987ad2018206d62e70590e41556525591cc98c34a: Status 404 returned error can't find the container with id 5c2bb9dca37002b38901ec7987ad2018206d62e70590e41556525591cc98c34a
Sep 16 19:18:27.087026 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:27.086985    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:27.447468 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:27.447362    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8" event={"ID":"3f10903e-be9d-4be6-a530-7aa76cd894e1","Type":"ContainerStarted","Data":"e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028"}
Sep 16 19:18:27.447468 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:27.447424    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8" event={"ID":"3f10903e-be9d-4be6-a530-7aa76cd894e1","Type":"ContainerStarted","Data":"5c2bb9dca37002b38901ec7987ad2018206d62e70590e41556525591cc98c34a"}
Sep 16 19:18:27.468958 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:27.468900    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8" podStartSLOduration=2.468865912 podStartE2EDuration="2.468865912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:27.46779467 +0000 UTC m=+2322.992253504" watchObservedRunningTime="2026-09-16 19:18:27.468865912 +0000 UTC m=+2322.993324777"
Sep 16 19:18:28.340874 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.340834    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:28.436395 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.436364    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"]
Sep 16 19:18:28.439673 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.439656    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.441981 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.441959    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 16 19:18:28.452253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.452230    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"]
Sep 16 19:18:28.457048 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.455345    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:28.541239 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.541201    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q74h\" (UniqueName: \"kubernetes.io/projected/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-kube-api-access-5q74h\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.541425 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.541264    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.541425 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.541296    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.541425 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.541405    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.541535 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.541452    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-home\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.541535 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.541470    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-dshm\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.642676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.642576    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-dshm\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.642676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.642617    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q74h\" (UniqueName: \"kubernetes.io/projected/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-kube-api-access-5q74h\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.642676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.642651    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.642676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.642676    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.643059 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.642715    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.643059 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.642757    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-home\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.643132 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.643110    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.643185 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.643167    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.643222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.643196    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-home\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.645120 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.645096    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-dshm\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.645348 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.645328    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.652012 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.651988    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q74h\" (UniqueName: \"kubernetes.io/projected/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-kube-api-access-5q74h\") pod \"roll-v2-kserve-d4dbf9d67-4vs79\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:28.750294 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:28.750258    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:29.034815 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:29.034773    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"]
Sep 16 19:18:29.038494 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:18:29.038469    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adb1bab_22bc_4f9d_9abe_5b4e9cc2c3d8.slice/crio-7343e1adc2226863709fcaf5d393d033fcf23398c8850c70446ddae128bb8794 WatchSource:0}: Error finding container 7343e1adc2226863709fcaf5d393d033fcf23398c8850c70446ddae128bb8794: Status 404 returned error can't find the container with id 7343e1adc2226863709fcaf5d393d033fcf23398c8850c70446ddae128bb8794
Sep 16 19:18:29.459355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:29.459252    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79" event={"ID":"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8","Type":"ContainerStarted","Data":"ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677"}
Sep 16 19:18:29.459355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:29.459315    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79" event={"ID":"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8","Type":"ContainerStarted","Data":"7343e1adc2226863709fcaf5d393d033fcf23398c8850c70446ddae128bb8794"}
Sep 16 19:18:29.487812 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:29.487761    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79" podStartSLOduration=1.487742902 podStartE2EDuration="1.487742902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:29.487225788 +0000 UTC m=+2325.011684625" watchObservedRunningTime="2026-09-16 19:18:29.487742902 +0000 UTC m=+2325.012201736"
Sep 16 19:18:29.751098 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:29.751002    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:36.341566 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:36.341471    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:36.353947 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:36.353921    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:36.499005 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:36.498977    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:18:37.705662 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:37.705627    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:37.816367 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:37.816329    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:37.899417 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:37.899381    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:38.751082 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:38.751049    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:38.764017 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:38.763990    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:39.519727 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:39.519699    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:18:45.145112 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:45.145073    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:51.422328 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:51.422292    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:51.497726 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:51.497687    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:55.464239 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.464206    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"]
Sep 16 19:18:55.467417 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.467392    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.469975 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.469944    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-9mtxm\""
Sep 16 19:18:55.470102 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.469994    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:18:55.470960 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.470939    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:18:55.471085 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.471050    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:18:55.478219 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.478185    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"]
Sep 16 19:18:55.506760 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.506726    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:55.575262 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.575223    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.575444 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.575267    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.575444 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.575355    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.575444 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.575406    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.575444 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.575431    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-768pr\" (UniqueName: \"kubernetes.io/projected/851d2d21-e58c-4d6b-9ec7-1d401ca73035-kube-api-access-768pr\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.575629 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.575488    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.676100 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.676061    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.676302 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.676106    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.676302 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.676142    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.676302 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.676166    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-768pr\" (UniqueName: \"kubernetes.io/projected/851d2d21-e58c-4d6b-9ec7-1d401ca73035-kube-api-access-768pr\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.676302 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.676217    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.676302 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.676236    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.676680 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.676651    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.676755 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.676690    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.676755 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.676666    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.678817 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.678790    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.678817 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.678800    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.685996 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.685974    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-768pr\" (UniqueName: \"kubernetes.io/projected/851d2d21-e58c-4d6b-9ec7-1d401ca73035-kube-api-access-768pr\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.778144 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.778104    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:18:55.799249 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:55.799221    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:56.051962 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:56.051938    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"]
Sep 16 19:18:56.054165 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:18:56.054134    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851d2d21_e58c_4d6b_9ec7_1d401ca73035.slice/crio-edd78910b2dcdb8bb1875019ae97775d6848a0f3a77c2a14ede337f7bed69994 WatchSource:0}: Error finding container edd78910b2dcdb8bb1875019ae97775d6848a0f3a77c2a14ede337f7bed69994: Status 404 returned error can't find the container with id edd78910b2dcdb8bb1875019ae97775d6848a0f3a77c2a14ede337f7bed69994
Sep 16 19:18:56.586793 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:56.586746    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4" event={"ID":"851d2d21-e58c-4d6b-9ec7-1d401ca73035","Type":"ContainerStarted","Data":"bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88"}
Sep 16 19:18:56.587282 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:56.586818    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4" event={"ID":"851d2d21-e58c-4d6b-9ec7-1d401ca73035","Type":"ContainerStarted","Data":"edd78910b2dcdb8bb1875019ae97775d6848a0f3a77c2a14ede337f7bed69994"}
Sep 16 19:18:56.611672 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:56.611600    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4" podStartSLOduration=1.611578148 podStartE2EDuration="1.611578148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:56.610508447 +0000 UTC m=+2352.134967282" watchObservedRunningTime="2026-09-16 19:18:56.611578148 +0000 UTC m=+2352.136036986"
Sep 16 19:18:56.778518 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:18:56.778484    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:19:04.799309 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:04.799265    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:05.778544 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:05.778507    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:19:05.791096 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:05.791064    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:19:06.647539 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:06.647507    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:19:10.756603 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:10.756565    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:10.856380 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:10.856346    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:11.599092 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:11.599046    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:24.363769 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:24.363733    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:29.321074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.321038    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:29.361669 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.361634    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"]
Sep 16 19:19:29.371176 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:19:29.371139    2590 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:19:29.371176 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:19:29.371176 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:19:29.371176 ip-10-0-143-44 kubenswrapper[2590]:  > containerID="bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88" cmd=["/bin/sleep","15"]
Sep 16 19:19:29.371417 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.371193    2590 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4" message="" err=<
Sep 16 19:19:29.371417 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:19:29.371417 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:19:29.371417 ip-10-0-143-44 kubenswrapper[2590]:  >
Sep 16 19:19:29.371417 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:19:29.371205    2590 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:19:29.371417 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:19:29.371417 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:19:29.371417 ip-10-0-143-44 kubenswrapper[2590]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4" podUID="851d2d21-e58c-4d6b-9ec7-1d401ca73035" containerName="main" containerID="cri-o://bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88"
Sep 16 19:19:29.371417 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.371259    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4" podUID="851d2d21-e58c-4d6b-9ec7-1d401ca73035" containerName="main" containerID="cri-o://bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88" gracePeriod=60
Sep 16 19:19:29.401213 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.401170    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:29.611682 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.611658    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:19:29.664003 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.663969    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tls-certs\") pod \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") "
Sep 16 19:19:29.664172 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.664040    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-768pr\" (UniqueName: \"kubernetes.io/projected/851d2d21-e58c-4d6b-9ec7-1d401ca73035-kube-api-access-768pr\") pod \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") "
Sep 16 19:19:29.664172 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.664075    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-dshm\") pod \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") "
Sep 16 19:19:29.664172 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.664123    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tmp-dir\") pod \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") "
Sep 16 19:19:29.664172 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.664168    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-home\") pod \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") "
Sep 16 19:19:29.664381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.664192    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-model-cache\") pod \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\" (UID: \"851d2d21-e58c-4d6b-9ec7-1d401ca73035\") "
Sep 16 19:19:29.664509 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.664460    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-home" (OuterVolumeSpecName: "home") pod "851d2d21-e58c-4d6b-9ec7-1d401ca73035" (UID: "851d2d21-e58c-4d6b-9ec7-1d401ca73035"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.664641 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.664518    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "851d2d21-e58c-4d6b-9ec7-1d401ca73035" (UID: "851d2d21-e58c-4d6b-9ec7-1d401ca73035"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.664641 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.664536    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-model-cache" (OuterVolumeSpecName: "model-cache") pod "851d2d21-e58c-4d6b-9ec7-1d401ca73035" (UID: "851d2d21-e58c-4d6b-9ec7-1d401ca73035"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.666585 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.666563    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851d2d21-e58c-4d6b-9ec7-1d401ca73035-kube-api-access-768pr" (OuterVolumeSpecName: "kube-api-access-768pr") pod "851d2d21-e58c-4d6b-9ec7-1d401ca73035" (UID: "851d2d21-e58c-4d6b-9ec7-1d401ca73035"). InnerVolumeSpecName "kube-api-access-768pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:29.666668 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.666581    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "851d2d21-e58c-4d6b-9ec7-1d401ca73035" (UID: "851d2d21-e58c-4d6b-9ec7-1d401ca73035"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:29.666668 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.666610    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-dshm" (OuterVolumeSpecName: "dshm") pod "851d2d21-e58c-4d6b-9ec7-1d401ca73035" (UID: "851d2d21-e58c-4d6b-9ec7-1d401ca73035"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.737820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.737773    2590 generic.go:356] "Generic (PLEG): container finished" podID="851d2d21-e58c-4d6b-9ec7-1d401ca73035" containerID="bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88" exitCode=0
Sep 16 19:19:29.737820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.737824    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"
Sep 16 19:19:29.738115 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.737861    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4" event={"ID":"851d2d21-e58c-4d6b-9ec7-1d401ca73035","Type":"ContainerDied","Data":"bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88"}
Sep 16 19:19:29.738115 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.737941    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4" event={"ID":"851d2d21-e58c-4d6b-9ec7-1d401ca73035","Type":"ContainerDied","Data":"edd78910b2dcdb8bb1875019ae97775d6848a0f3a77c2a14ede337f7bed69994"}
Sep 16 19:19:29.738115 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.737970    2590 scope.go:117] "RemoveContainer" containerID="bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88"
Sep 16 19:19:29.747168 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.747152    2590 scope.go:117] "RemoveContainer" containerID="bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88"
Sep 16 19:19:29.747456 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:19:29.747436    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88\": container with ID starting with bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88 not found: ID does not exist" containerID="bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88"
Sep 16 19:19:29.747520 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.747464    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88"} err="failed to get container status \"bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88\": rpc error: code = NotFound desc = could not find container \"bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88\": container with ID starting with bc7537629780b97462a23fc3161a7384a6fe182eda74bb65cb1993ff07a6df88 not found: ID does not exist"
Sep 16 19:19:29.762744 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.762717    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"]
Sep 16 19:19:29.765308 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.765290    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-768pr\" (UniqueName: \"kubernetes.io/projected/851d2d21-e58c-4d6b-9ec7-1d401ca73035-kube-api-access-768pr\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.765372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.765311    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.765372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.765322    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.765372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.765358    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.765372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.765366    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d2d21-e58c-4d6b-9ec7-1d401ca73035-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.765534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.765376    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d2d21-e58c-4d6b-9ec7-1d401ca73035-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.767405 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:29.767386    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lmlz4"]
Sep 16 19:19:31.025318 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:19:31.025281    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851d2d21-e58c-4d6b-9ec7-1d401ca73035" path="/var/lib/kubelet/pods/851d2d21-e58c-4d6b-9ec7-1d401ca73035/volumes"
Sep 16 19:20:00.099625 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:00.099584    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:00.128671 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:00.128635    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"]
Sep 16 19:20:00.195747 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:00.195717    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:00.362572 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:00.362484    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"]
Sep 16 19:20:00.393284 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:00.393253    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:02.654949 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.654902    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"]
Sep 16 19:20:02.655433 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.655250    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851d2d21-e58c-4d6b-9ec7-1d401ca73035" containerName="main"
Sep 16 19:20:02.655433 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.655262    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="851d2d21-e58c-4d6b-9ec7-1d401ca73035" containerName="main"
Sep 16 19:20:02.655433 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.655319    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="851d2d21-e58c-4d6b-9ec7-1d401ca73035" containerName="main"
Sep 16 19:20:02.658315 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.658290    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.662600 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.662573    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-model-cache\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.662600 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.662602    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.662768 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.662696    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.662768 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.662738    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-home\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.662768 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.662764    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-dshm\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.662915 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.662795    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mbv\" (UniqueName: \"kubernetes.io/projected/8ab8982e-0585-4b9e-a2d1-08c212ee1240-kube-api-access-d8mbv\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.666409 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.666389    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-dj2tk\""
Sep 16 19:20:02.667400 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.667378    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 16 19:20:02.667497 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.667386    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 16 19:20:02.667497 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.667422    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 16 19:20:02.673390 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.673359    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"]
Sep 16 19:20:02.688468 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.688441    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:02.763325 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.763288    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.763545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.763336    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-home\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.763545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.763364    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-dshm\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.763545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.763385    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mbv\" (UniqueName: \"kubernetes.io/projected/8ab8982e-0585-4b9e-a2d1-08c212ee1240-kube-api-access-d8mbv\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.763545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.763439    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-model-cache\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.763545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.763459    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.763815 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.763749    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.763815 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.763788    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-home\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.763965 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.763828    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-model-cache\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.765849 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.765826    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-dshm\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.766215 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.766197    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.774324 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.774295    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mbv\" (UniqueName: \"kubernetes.io/projected/8ab8982e-0585-4b9e-a2d1-08c212ee1240-kube-api-access-d8mbv\") pod \"owner-v1-kserve-75579ff5bc-k7g4r\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:02.969094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:02.968997    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:03.264972 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:20:03.264935    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab8982e_0585_4b9e_a2d1_08c212ee1240.slice/crio-8577f1d3a168043cfe784be912842e548b110fadd82fe38a8b0a018c4e4a70cc WatchSource:0}: Error finding container 8577f1d3a168043cfe784be912842e548b110fadd82fe38a8b0a018c4e4a70cc: Status 404 returned error can't find the container with id 8577f1d3a168043cfe784be912842e548b110fadd82fe38a8b0a018c4e4a70cc
Sep 16 19:20:03.265104 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:03.264986    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"]
Sep 16 19:20:03.890337 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:03.890296    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r" event={"ID":"8ab8982e-0585-4b9e-a2d1-08c212ee1240","Type":"ContainerStarted","Data":"e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996"}
Sep 16 19:20:03.890841 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:03.890362    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r" event={"ID":"8ab8982e-0585-4b9e-a2d1-08c212ee1240","Type":"ContainerStarted","Data":"8577f1d3a168043cfe784be912842e548b110fadd82fe38a8b0a018c4e4a70cc"}
Sep 16 19:20:03.914665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:03.914607    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r" podStartSLOduration=1.914590045 podStartE2EDuration="1.914590045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:03.913358249 +0000 UTC m=+2419.437817083" watchObservedRunningTime="2026-09-16 19:20:03.914590045 +0000 UTC m=+2419.439048879"
Sep 16 19:20:03.970059 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:03.970019    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:05.132928 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:05.132874    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:11.443741 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.443663    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"]
Sep 16 19:20:11.446836 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.446817    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.449421 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.449396    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 16 19:20:11.460251 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.460220    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"]
Sep 16 19:20:11.486446 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.486406    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:11.523447 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.523413    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-home\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.523640 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.523461    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-model-cache\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.523640 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.523526    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.523640 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.523576    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-dshm\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.523640 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.523593    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03840ee-7139-496f-869e-d70f22f8d522-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.523838 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.523662    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjgzn\" (UniqueName: \"kubernetes.io/projected/a03840ee-7139-496f-869e-d70f22f8d522-kube-api-access-vjgzn\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.624173 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.624137    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-home\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.624374 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.624190    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-model-cache\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.624374 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.624220    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.624374 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.624245    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-dshm\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.624374 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.624364    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03840ee-7139-496f-869e-d70f22f8d522-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.624606 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.624418    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgzn\" (UniqueName: \"kubernetes.io/projected/a03840ee-7139-496f-869e-d70f22f8d522-kube-api-access-vjgzn\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.624606 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.624587    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-home\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.624715 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.624640    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.624715 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.624689    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-model-cache\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.627075 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.627038    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-dshm\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.627255 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.627236    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03840ee-7139-496f-869e-d70f22f8d522-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.633176 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.633152    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjgzn\" (UniqueName: \"kubernetes.io/projected/a03840ee-7139-496f-869e-d70f22f8d522-kube-api-access-vjgzn\") pod \"owner-v2-kserve-5cf8959f95-mbcn4\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:11.756998 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:11.756870    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:12.051256 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:12.048078    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"]
Sep 16 19:20:12.051256 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:20:12.050780    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03840ee_7139_496f_869e_d70f22f8d522.slice/crio-2318564a08d10f58072daffdaa18486eb096a82f8708f1ac37149868e0ec6cd7 WatchSource:0}: Error finding container 2318564a08d10f58072daffdaa18486eb096a82f8708f1ac37149868e0ec6cd7: Status 404 returned error can't find the container with id 2318564a08d10f58072daffdaa18486eb096a82f8708f1ac37149868e0ec6cd7
Sep 16 19:20:12.933535 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:12.933498    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4" event={"ID":"a03840ee-7139-496f-869e-d70f22f8d522","Type":"ContainerStarted","Data":"38aafbf39168792e6c2e1f830d77efd0b74099dbf636008f69d16cde66596e01"}
Sep 16 19:20:12.933975 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:12.933550    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4" event={"ID":"a03840ee-7139-496f-869e-d70f22f8d522","Type":"ContainerStarted","Data":"2318564a08d10f58072daffdaa18486eb096a82f8708f1ac37149868e0ec6cd7"}
Sep 16 19:20:12.953573 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:12.953523    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4" podStartSLOduration=1.953505201 podStartE2EDuration="1.953505201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:12.952551689 +0000 UTC m=+2428.477010524" watchObservedRunningTime="2026-09-16 19:20:12.953505201 +0000 UTC m=+2428.477964035"
Sep 16 19:20:12.969690 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:12.969651    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:12.982184 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:12.982156    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:13.757329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:13.757290    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:13.952113 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:13.952085    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:20:15.148184 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.148142    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79" podUID="0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" containerName="main" containerID="cri-o://ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677" gracePeriod=45
Sep 16 19:20:15.381503 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.381422    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8" podUID="3f10903e-be9d-4be6-a530-7aa76cd894e1" containerName="main" containerID="cri-o://e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028" gracePeriod=45
Sep 16 19:20:15.449386 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.449319    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:20:15.449740 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.449721    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-dshm\") pod \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") "
Sep 16 19:20:15.449843 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.449773    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tls-certs\") pod \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") "
Sep 16 19:20:15.449843 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.449820    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q74h\" (UniqueName: \"kubernetes.io/projected/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-kube-api-access-5q74h\") pod \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") "
Sep 16 19:20:15.450067 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.449855    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tmp-dir\") pod \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") "
Sep 16 19:20:15.450067 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.449909    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-model-cache\") pod \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") "
Sep 16 19:20:15.450067 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.449941    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-home\") pod \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\" (UID: \"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8\") "
Sep 16 19:20:15.450227 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.450203    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" (UID: "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:15.450270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.450219    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" (UID: "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:15.450270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.450235    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-home" (OuterVolumeSpecName: "home") pod "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" (UID: "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:15.452199 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.452175    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" (UID: "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:15.452199 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.452190    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-kube-api-access-5q74h" (OuterVolumeSpecName: "kube-api-access-5q74h") pod "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" (UID: "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8"). InnerVolumeSpecName "kube-api-access-5q74h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:15.452468 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.452219    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-dshm" (OuterVolumeSpecName: "dshm") pod "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" (UID: "0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:15.488940 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.488896    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8" podUID="3f10903e-be9d-4be6-a530-7aa76cd894e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 19:20:15.550673 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.550641    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.550673 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.550673    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.550673 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.550685    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q74h\" (UniqueName: \"kubernetes.io/projected/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-kube-api-access-5q74h\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.551006 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.550693    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.551006 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.550702    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.551006 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.550714    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.627100 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.627075    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:20:15.651194 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651157    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f10903e-be9d-4be6-a530-7aa76cd894e1-tls-certs\") pod \"3f10903e-be9d-4be6-a530-7aa76cd894e1\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") "
Sep 16 19:20:15.651359 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651239    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-tmp-dir\") pod \"3f10903e-be9d-4be6-a530-7aa76cd894e1\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") "
Sep 16 19:20:15.651359 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651283    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-dshm\") pod \"3f10903e-be9d-4be6-a530-7aa76cd894e1\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") "
Sep 16 19:20:15.651359 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651310    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gjmq\" (UniqueName: \"kubernetes.io/projected/3f10903e-be9d-4be6-a530-7aa76cd894e1-kube-api-access-8gjmq\") pod \"3f10903e-be9d-4be6-a530-7aa76cd894e1\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") "
Sep 16 19:20:15.651359 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651345    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-model-cache\") pod \"3f10903e-be9d-4be6-a530-7aa76cd894e1\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") "
Sep 16 19:20:15.651575 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651367    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-home\") pod \"3f10903e-be9d-4be6-a530-7aa76cd894e1\" (UID: \"3f10903e-be9d-4be6-a530-7aa76cd894e1\") "
Sep 16 19:20:15.651575 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651464    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f10903e-be9d-4be6-a530-7aa76cd894e1" (UID: "3f10903e-be9d-4be6-a530-7aa76cd894e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:15.651679 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651618    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f10903e-be9d-4be6-a530-7aa76cd894e1" (UID: "3f10903e-be9d-4be6-a530-7aa76cd894e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:15.651751 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651681    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-home" (OuterVolumeSpecName: "home") pod "3f10903e-be9d-4be6-a530-7aa76cd894e1" (UID: "3f10903e-be9d-4be6-a530-7aa76cd894e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:15.651751 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.651703    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.653513 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.653486    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-dshm" (OuterVolumeSpecName: "dshm") pod "3f10903e-be9d-4be6-a530-7aa76cd894e1" (UID: "3f10903e-be9d-4be6-a530-7aa76cd894e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:15.653623 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.653541    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f10903e-be9d-4be6-a530-7aa76cd894e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f10903e-be9d-4be6-a530-7aa76cd894e1" (UID: "3f10903e-be9d-4be6-a530-7aa76cd894e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:15.653623 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.653542    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f10903e-be9d-4be6-a530-7aa76cd894e1-kube-api-access-8gjmq" (OuterVolumeSpecName: "kube-api-access-8gjmq") pod "3f10903e-be9d-4be6-a530-7aa76cd894e1" (UID: "3f10903e-be9d-4be6-a530-7aa76cd894e1"). InnerVolumeSpecName "kube-api-access-8gjmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:15.753007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.752913    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.753007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.752945    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gjmq\" (UniqueName: \"kubernetes.io/projected/3f10903e-be9d-4be6-a530-7aa76cd894e1-kube-api-access-8gjmq\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.753007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.752956    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.753007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.752964    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f10903e-be9d-4be6-a530-7aa76cd894e1-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.753007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.752973    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f10903e-be9d-4be6-a530-7aa76cd894e1-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:20:15.952296 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.952254    2590 generic.go:356] "Generic (PLEG): container finished" podID="3f10903e-be9d-4be6-a530-7aa76cd894e1" containerID="e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028" exitCode=0
Sep 16 19:20:15.952296 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.952286    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"
Sep 16 19:20:15.952627 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.952333    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8" event={"ID":"3f10903e-be9d-4be6-a530-7aa76cd894e1","Type":"ContainerDied","Data":"e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028"}
Sep 16 19:20:15.952627 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.952390    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8" event={"ID":"3f10903e-be9d-4be6-a530-7aa76cd894e1","Type":"ContainerDied","Data":"5c2bb9dca37002b38901ec7987ad2018206d62e70590e41556525591cc98c34a"}
Sep 16 19:20:15.952627 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.952414    2590 scope.go:117] "RemoveContainer" containerID="e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028"
Sep 16 19:20:15.953793 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.953760    2590 generic.go:356] "Generic (PLEG): container finished" podID="0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" containerID="ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677" exitCode=0
Sep 16 19:20:15.953953 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.953806    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79" event={"ID":"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8","Type":"ContainerDied","Data":"ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677"}
Sep 16 19:20:15.953953 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.953823    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"
Sep 16 19:20:15.953953 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.953848    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79" event={"ID":"0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8","Type":"ContainerDied","Data":"7343e1adc2226863709fcaf5d393d033fcf23398c8850c70446ddae128bb8794"}
Sep 16 19:20:15.965603 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.965196    2590 scope.go:117] "RemoveContainer" containerID="e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028"
Sep 16 19:20:15.966694 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:20:15.966583    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028\": container with ID starting with e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028 not found: ID does not exist" containerID="e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028"
Sep 16 19:20:15.966694 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.966627    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028"} err="failed to get container status \"e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028\": rpc error: code = NotFound desc = could not find container \"e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028\": container with ID starting with e1ead4e1654e7db7ab9a9c06a7e5a6a470ab898c818c1cea33ddee11e22e6028 not found: ID does not exist"
Sep 16 19:20:15.966694 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.966658    2590 scope.go:117] "RemoveContainer" containerID="ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677"
Sep 16 19:20:15.976548 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.976527    2590 scope.go:117] "RemoveContainer" containerID="ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677"
Sep 16 19:20:15.976840 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:20:15.976820    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677\": container with ID starting with ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677 not found: ID does not exist" containerID="ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677"
Sep 16 19:20:15.977019 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.976848    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677"} err="failed to get container status \"ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677\": rpc error: code = NotFound desc = could not find container \"ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677\": container with ID starting with ba5cb9cb3cfb1b26488e90e708e041c9bf3b8c4c51293b99bd87e046221ac677 not found: ID does not exist"
Sep 16 19:20:15.981714 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.981691    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"]
Sep 16 19:20:15.985742 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.985721    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-78qh8"]
Sep 16 19:20:15.996359 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:15.996333    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"]
Sep 16 19:20:16.002111 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:16.002084    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4vs79"]
Sep 16 19:20:17.025997 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:17.025962    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" path="/var/lib/kubelet/pods/0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8/volumes"
Sep 16 19:20:17.026409 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:17.026349    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f10903e-be9d-4be6-a530-7aa76cd894e1" path="/var/lib/kubelet/pods/3f10903e-be9d-4be6-a530-7aa76cd894e1/volumes"
Sep 16 19:20:18.664311 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:18.664277    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:18.757102 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:18.757066    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:21.757742 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:21.757704    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:21.770256 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:21.770230    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:21.993301 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:21.993267    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:20:25.358611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:25.358571    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:32.916115 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:32.916061    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:33.014294 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:33.014253    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:44.803659 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:44.803625    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:51.334848 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.334814    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:51.437506 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.437473    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:51.978126 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.978089    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"]
Sep 16 19:20:51.978477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.978381    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f10903e-be9d-4be6-a530-7aa76cd894e1" containerName="main"
Sep 16 19:20:51.978477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.978392    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f10903e-be9d-4be6-a530-7aa76cd894e1" containerName="main"
Sep 16 19:20:51.978477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.978402    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" containerName="main"
Sep 16 19:20:51.978477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.978408    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" containerName="main"
Sep 16 19:20:51.978477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.978459    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f10903e-be9d-4be6-a530-7aa76cd894e1" containerName="main"
Sep 16 19:20:51.978477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.978469    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="0adb1bab-22bc-4f9d-9abe-5b4e9cc2c3d8" containerName="main"
Sep 16 19:20:51.981264 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.981241    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:51.985832 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.985807    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:20:51.986051 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.985826    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:20:51.986051 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.985846    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:20:51.986051 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:51.985808    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-vhfcs\""
Sep 16 19:20:52.008134 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.008103    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"]
Sep 16 19:20:52.070984 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.070947    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:52.163264 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.163220    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.163264 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.163270    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.163509 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.163324    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.163509 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.163409    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea67604-a010-4059-b094-087449c37651-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.163509 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.163435    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hszn\" (UniqueName: \"kubernetes.io/projected/eea67604-a010-4059-b094-087449c37651-kube-api-access-7hszn\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.163648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.163505    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.264355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.264272    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.264355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.264309    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.264551 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.264438    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.264551 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.264502    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea67604-a010-4059-b094-087449c37651-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.264551 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.264526    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hszn\" (UniqueName: \"kubernetes.io/projected/eea67604-a010-4059-b094-087449c37651-kube-api-access-7hszn\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.264712 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.264564    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.264712 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.264660    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.264828 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.264799    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.264910 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.264856    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.266801 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.266781    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.267176 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.267157    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea67604-a010-4059-b094-087449c37651-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.286491 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.286466    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hszn\" (UniqueName: \"kubernetes.io/projected/eea67604-a010-4059-b094-087449c37651-kube-api-access-7hszn\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv\" (UID: \"eea67604-a010-4059-b094-087449c37651\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.294347 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.294319    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:20:52.344105 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.344063    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:52.576604 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:52.576577    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"]
Sep 16 19:20:52.578820 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:20:52.578794    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea67604_a010_4059_b094_087449c37651.slice/crio-13a6a1ccfd778718860d993a1a2d499d0297cda2aa6d61b79267044a8a33e263 WatchSource:0}: Error finding container 13a6a1ccfd778718860d993a1a2d499d0297cda2aa6d61b79267044a8a33e263: Status 404 returned error can't find the container with id 13a6a1ccfd778718860d993a1a2d499d0297cda2aa6d61b79267044a8a33e263
Sep 16 19:20:53.123226 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:53.123194    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv" event={"ID":"eea67604-a010-4059-b094-087449c37651","Type":"ContainerStarted","Data":"81d4303d824cae7488cf6eee98d1741866a09627feb9daf688caeb0b34ae0f5e"}
Sep 16 19:20:53.123446 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:53.123258    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv" event={"ID":"eea67604-a010-4059-b094-087449c37651","Type":"ContainerStarted","Data":"13a6a1ccfd778718860d993a1a2d499d0297cda2aa6d61b79267044a8a33e263"}
Sep 16 19:20:53.163319 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:53.163263    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv" podStartSLOduration=2.163245106 podStartE2EDuration="2.163245106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:53.161777485 +0000 UTC m=+2468.686236311" watchObservedRunningTime="2026-09-16 19:20:53.163245106 +0000 UTC m=+2468.687703940"
Sep 16 19:20:53.295450 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:20:53.295359    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:21:02.295518 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:02.295476    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:21:02.308649 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:02.308617    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:21:03.178903 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:03.178856    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:21:04.843357 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:04.843309    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:08.164571 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:08.164524    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:10.822564 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:10.822532    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"]
Sep 16 19:21:10.916140 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:10.916109    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:25.842081 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:25.842026    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r" podUID="8ab8982e-0585-4b9e-a2d1-08c212ee1240" containerName="main" containerID="cri-o://e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996" gracePeriod=45
Sep 16 19:21:25.941120 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:25.941075    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r" podUID="8ab8982e-0585-4b9e-a2d1-08c212ee1240" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 16 19:21:26.094568 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.094509    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:21:26.137189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137155    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8mbv\" (UniqueName: \"kubernetes.io/projected/8ab8982e-0585-4b9e-a2d1-08c212ee1240-kube-api-access-d8mbv\") pod \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") "
Sep 16 19:21:26.137381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137202    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-model-cache\") pod \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") "
Sep 16 19:21:26.137381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137241    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-home\") pod \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") "
Sep 16 19:21:26.137381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137271    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tmp-dir\") pod \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") "
Sep 16 19:21:26.137381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137300    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tls-certs\") pod \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") "
Sep 16 19:21:26.137381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137357    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-dshm\") pod \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\" (UID: \"8ab8982e-0585-4b9e-a2d1-08c212ee1240\") "
Sep 16 19:21:26.137667 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137536    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ab8982e-0585-4b9e-a2d1-08c212ee1240" (UID: "8ab8982e-0585-4b9e-a2d1-08c212ee1240"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.137667 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137562    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-home" (OuterVolumeSpecName: "home") pod "8ab8982e-0585-4b9e-a2d1-08c212ee1240" (UID: "8ab8982e-0585-4b9e-a2d1-08c212ee1240"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.137667 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137608    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ab8982e-0585-4b9e-a2d1-08c212ee1240" (UID: "8ab8982e-0585-4b9e-a2d1-08c212ee1240"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.137827 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137673    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.137827 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137691    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.137827 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.137704    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.139829 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.139791    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ab8982e-0585-4b9e-a2d1-08c212ee1240" (UID: "8ab8982e-0585-4b9e-a2d1-08c212ee1240"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:26.139829 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.139810    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-dshm" (OuterVolumeSpecName: "dshm") pod "8ab8982e-0585-4b9e-a2d1-08c212ee1240" (UID: "8ab8982e-0585-4b9e-a2d1-08c212ee1240"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.140008 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.139987    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab8982e-0585-4b9e-a2d1-08c212ee1240-kube-api-access-d8mbv" (OuterVolumeSpecName: "kube-api-access-d8mbv") pod "8ab8982e-0585-4b9e-a2d1-08c212ee1240" (UID: "8ab8982e-0585-4b9e-a2d1-08c212ee1240"). InnerVolumeSpecName "kube-api-access-d8mbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:26.238566 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.238529    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab8982e-0585-4b9e-a2d1-08c212ee1240-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.238566 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.238561    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab8982e-0585-4b9e-a2d1-08c212ee1240-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.238566 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.238571    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8mbv\" (UniqueName: \"kubernetes.io/projected/8ab8982e-0585-4b9e-a2d1-08c212ee1240-kube-api-access-d8mbv\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.271722 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.271689    2590 generic.go:356] "Generic (PLEG): container finished" podID="8ab8982e-0585-4b9e-a2d1-08c212ee1240" containerID="e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996" exitCode=0
Sep 16 19:21:26.271923 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.271736    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"
Sep 16 19:21:26.271923 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.271743    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r" event={"ID":"8ab8982e-0585-4b9e-a2d1-08c212ee1240","Type":"ContainerDied","Data":"e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996"}
Sep 16 19:21:26.271923 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.271787    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r" event={"ID":"8ab8982e-0585-4b9e-a2d1-08c212ee1240","Type":"ContainerDied","Data":"8577f1d3a168043cfe784be912842e548b110fadd82fe38a8b0a018c4e4a70cc"}
Sep 16 19:21:26.271923 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.271808    2590 scope.go:117] "RemoveContainer" containerID="e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996"
Sep 16 19:21:26.281186 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.281164    2590 scope.go:117] "RemoveContainer" containerID="e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996"
Sep 16 19:21:26.281463 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:21:26.281446    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996\": container with ID starting with e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996 not found: ID does not exist" containerID="e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996"
Sep 16 19:21:26.281514 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.281471    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996"} err="failed to get container status \"e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996\": rpc error: code = NotFound desc = could not find container \"e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996\": container with ID starting with e66e9a421da30dd6429f2c62f8e40bedf3b456fc729155911802677d75ac4996 not found: ID does not exist"
Sep 16 19:21:26.296007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.295970    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"]
Sep 16 19:21:26.300614 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:26.300590    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-k7g4r"]
Sep 16 19:21:27.025893 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:27.025850    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab8982e-0585-4b9e-a2d1-08c212ee1240" path="/var/lib/kubelet/pods/8ab8982e-0585-4b9e-a2d1-08c212ee1240/volumes"
Sep 16 19:21:31.100976 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:31.100934    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:32.589780 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:32.589745    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:35.309662 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:35.309615    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:36.172938 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:36.172852    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"]
Sep 16 19:21:36.260811 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:36.260773    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:49.944936 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:49.944860    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:50.220454 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.220364    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"]
Sep 16 19:21:50.220699 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.220685    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab8982e-0585-4b9e-a2d1-08c212ee1240" containerName="main"
Sep 16 19:21:50.220744 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.220702    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab8982e-0585-4b9e-a2d1-08c212ee1240" containerName="main"
Sep 16 19:21:50.220779 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.220762    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab8982e-0585-4b9e-a2d1-08c212ee1240" containerName="main"
Sep 16 19:21:50.228270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.228252    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:21:50.230871 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.230849    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-lzvcz\""
Sep 16 19:21:50.231744 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.231725    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:21:50.264012 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.263980    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"]
Sep 16 19:21:50.268614 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.268584    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:50.347173 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.347126    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l559s\" (UniqueName: \"kubernetes.io/projected/7e630b67-ad9a-4b4d-8f64-638392dae017-kube-api-access-l559s\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992\" (UID: \"7e630b67-ad9a-4b4d-8f64-638392dae017\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:21:50.347367 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.347183    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e630b67-ad9a-4b4d-8f64-638392dae017-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992\" (UID: \"7e630b67-ad9a-4b4d-8f64-638392dae017\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:21:50.448134 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.448095    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l559s\" (UniqueName: \"kubernetes.io/projected/7e630b67-ad9a-4b4d-8f64-638392dae017-kube-api-access-l559s\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992\" (UID: \"7e630b67-ad9a-4b4d-8f64-638392dae017\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:21:50.448321 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.448142    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e630b67-ad9a-4b4d-8f64-638392dae017-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992\" (UID: \"7e630b67-ad9a-4b4d-8f64-638392dae017\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:21:50.450935 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.450865    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e630b67-ad9a-4b4d-8f64-638392dae017-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992\" (UID: \"7e630b67-ad9a-4b4d-8f64-638392dae017\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:21:50.459536 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.459508    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l559s\" (UniqueName: \"kubernetes.io/projected/7e630b67-ad9a-4b4d-8f64-638392dae017-kube-api-access-l559s\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992\" (UID: \"7e630b67-ad9a-4b4d-8f64-638392dae017\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:21:50.538697 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.538660    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:21:50.822686 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:50.822655    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"]
Sep 16 19:21:50.824091 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:21:50.824065    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e630b67_ad9a_4b4d_8f64_638392dae017.slice/crio-c75433bb6c4376d024cfc1cd029d7a89b852f51f6bd26e1c4e199351210fd636 WatchSource:0}: Error finding container c75433bb6c4376d024cfc1cd029d7a89b852f51f6bd26e1c4e199351210fd636: Status 404 returned error can't find the container with id c75433bb6c4376d024cfc1cd029d7a89b852f51f6bd26e1c4e199351210fd636
Sep 16 19:21:51.192549 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.192424    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4" podUID="a03840ee-7139-496f-869e-d70f22f8d522" containerName="main" containerID="cri-o://38aafbf39168792e6c2e1f830d77efd0b74099dbf636008f69d16cde66596e01" gracePeriod=45
Sep 16 19:21:51.380629 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.380574    2590 generic.go:356] "Generic (PLEG): container finished" podID="a03840ee-7139-496f-869e-d70f22f8d522" containerID="38aafbf39168792e6c2e1f830d77efd0b74099dbf636008f69d16cde66596e01" exitCode=0
Sep 16 19:21:51.380813 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.380647    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4" event={"ID":"a03840ee-7139-496f-869e-d70f22f8d522","Type":"ContainerDied","Data":"38aafbf39168792e6c2e1f830d77efd0b74099dbf636008f69d16cde66596e01"}
Sep 16 19:21:51.382582 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.382556    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" event={"ID":"7e630b67-ad9a-4b4d-8f64-638392dae017","Type":"ContainerStarted","Data":"d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd"}
Sep 16 19:21:51.382730 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.382623    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" event={"ID":"7e630b67-ad9a-4b4d-8f64-638392dae017","Type":"ContainerStarted","Data":"c75433bb6c4376d024cfc1cd029d7a89b852f51f6bd26e1c4e199351210fd636"}
Sep 16 19:21:51.382795 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.382781    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:21:51.412326 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.412271    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" podStartSLOduration=1.412253256 podStartE2EDuration="1.412253256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:51.410954898 +0000 UTC m=+2526.935413735" watchObservedRunningTime="2026-09-16 19:21:51.412253256 +0000 UTC m=+2526.936712090"
Sep 16 19:21:51.425978 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.425951    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:21:51.558529 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.558498    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjgzn\" (UniqueName: \"kubernetes.io/projected/a03840ee-7139-496f-869e-d70f22f8d522-kube-api-access-vjgzn\") pod \"a03840ee-7139-496f-869e-d70f22f8d522\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") "
Sep 16 19:21:51.558529 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.558537    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03840ee-7139-496f-869e-d70f22f8d522-tls-certs\") pod \"a03840ee-7139-496f-869e-d70f22f8d522\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") "
Sep 16 19:21:51.558758 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.558571    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-dshm\") pod \"a03840ee-7139-496f-869e-d70f22f8d522\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") "
Sep 16 19:21:51.558758 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.558601    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-model-cache\") pod \"a03840ee-7139-496f-869e-d70f22f8d522\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") "
Sep 16 19:21:51.558960 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.558792    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-home\") pod \"a03840ee-7139-496f-869e-d70f22f8d522\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") "
Sep 16 19:21:51.558960 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.558836    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-model-cache" (OuterVolumeSpecName: "model-cache") pod "a03840ee-7139-496f-869e-d70f22f8d522" (UID: "a03840ee-7139-496f-869e-d70f22f8d522"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:51.558960 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.558915    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-tmp-dir\") pod \"a03840ee-7139-496f-869e-d70f22f8d522\" (UID: \"a03840ee-7139-496f-869e-d70f22f8d522\") "
Sep 16 19:21:51.559105 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.559033    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-home" (OuterVolumeSpecName: "home") pod "a03840ee-7139-496f-869e-d70f22f8d522" (UID: "a03840ee-7139-496f-869e-d70f22f8d522"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:51.559105 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.559080    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a03840ee-7139-496f-869e-d70f22f8d522" (UID: "a03840ee-7139-496f-869e-d70f22f8d522"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:51.559272 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.559252    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:51.559349 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.559278    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:51.559548 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.559365    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:51.560901 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.560864    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-dshm" (OuterVolumeSpecName: "dshm") pod "a03840ee-7139-496f-869e-d70f22f8d522" (UID: "a03840ee-7139-496f-869e-d70f22f8d522"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:51.561001 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.560982    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03840ee-7139-496f-869e-d70f22f8d522-kube-api-access-vjgzn" (OuterVolumeSpecName: "kube-api-access-vjgzn") pod "a03840ee-7139-496f-869e-d70f22f8d522" (UID: "a03840ee-7139-496f-869e-d70f22f8d522"). InnerVolumeSpecName "kube-api-access-vjgzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:51.561092 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.561078    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03840ee-7139-496f-869e-d70f22f8d522-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a03840ee-7139-496f-869e-d70f22f8d522" (UID: "a03840ee-7139-496f-869e-d70f22f8d522"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:51.660796 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.660756    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjgzn\" (UniqueName: \"kubernetes.io/projected/a03840ee-7139-496f-869e-d70f22f8d522-kube-api-access-vjgzn\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:51.660796 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.660792    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03840ee-7139-496f-869e-d70f22f8d522-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:51.661030 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:51.660808    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03840ee-7139-496f-869e-d70f22f8d522-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:52.388502 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:52.388471    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"
Sep 16 19:21:52.388502 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:52.388488    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4" event={"ID":"a03840ee-7139-496f-869e-d70f22f8d522","Type":"ContainerDied","Data":"2318564a08d10f58072daffdaa18486eb096a82f8708f1ac37149868e0ec6cd7"}
Sep 16 19:21:52.389102 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:52.388549    2590 scope.go:117] "RemoveContainer" containerID="38aafbf39168792e6c2e1f830d77efd0b74099dbf636008f69d16cde66596e01"
Sep 16 19:21:52.433202 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:52.433166    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"]
Sep 16 19:21:52.434396 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:52.434372    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mbcn4"]
Sep 16 19:21:53.025121 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:53.025088    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a03840ee-7139-496f-869e-d70f22f8d522" path="/var/lib/kubelet/pods/a03840ee-7139-496f-869e-d70f22f8d522/volumes"
Sep 16 19:21:53.332363 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:21:53.332281    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:05.313560 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:05.313507    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:13.728492 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:13.728451    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:16.722114 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:16.722084    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:20.098943 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:20.098908    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:20.175104 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:20.175070    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:22.391000 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:22.390964    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:22:28.380248 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:28.380212    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"]
Sep 16 19:22:28.391330 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:28.391291    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:28.488171 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:28.488139    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:29.466181 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:29.466147    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:32.390408 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:32.390373    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:32.974018 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:32.973985    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:33.083699 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:33.083665    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:38.214723 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:38.214688    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:41.924194 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:41.924160    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:42.228300 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.228227    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"]
Sep 16 19:22:42.228527 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.228515    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03840ee-7139-496f-869e-d70f22f8d522" containerName="main"
Sep 16 19:22:42.228573 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.228528    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03840ee-7139-496f-869e-d70f22f8d522" containerName="main"
Sep 16 19:22:42.228608 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.228574    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="a03840ee-7139-496f-869e-d70f22f8d522" containerName="main"
Sep 16 19:22:42.231554 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.231537    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:22:42.237130 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.237103    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:22:42.237246 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.237108    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-2j8l2\""
Sep 16 19:22:42.237973 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.237933    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:22:42.239143 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.239125    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-2qtc9\""
Sep 16 19:22:42.240369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.240351    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:22:42.262105 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.262077    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"]
Sep 16 19:22:42.271444 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.271412    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:42.384158 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.384123    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/689b0d75-f6dc-4f06-b801-d8de1a801e73-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd\" (UID: \"689b0d75-f6dc-4f06-b801-d8de1a801e73\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:22:42.384356 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.384170    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbj2j\" (UniqueName: \"kubernetes.io/projected/689b0d75-f6dc-4f06-b801-d8de1a801e73-kube-api-access-fbj2j\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd\" (UID: \"689b0d75-f6dc-4f06-b801-d8de1a801e73\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:22:42.390262 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.390227    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:42.485168 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.485070    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbj2j\" (UniqueName: \"kubernetes.io/projected/689b0d75-f6dc-4f06-b801-d8de1a801e73-kube-api-access-fbj2j\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd\" (UID: \"689b0d75-f6dc-4f06-b801-d8de1a801e73\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:22:42.485168 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.485164    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/689b0d75-f6dc-4f06-b801-d8de1a801e73-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd\" (UID: \"689b0d75-f6dc-4f06-b801-d8de1a801e73\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:22:42.487867 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.487834    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/689b0d75-f6dc-4f06-b801-d8de1a801e73-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd\" (UID: \"689b0d75-f6dc-4f06-b801-d8de1a801e73\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:22:42.509454 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.509415    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbj2j\" (UniqueName: \"kubernetes.io/projected/689b0d75-f6dc-4f06-b801-d8de1a801e73-kube-api-access-fbj2j\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd\" (UID: \"689b0d75-f6dc-4f06-b801-d8de1a801e73\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:22:42.541442 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.541407    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:22:42.830228 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:42.830196    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"]
Sep 16 19:22:42.832477 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:22:42.832448    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689b0d75_f6dc_4f06_b801_d8de1a801e73.slice/crio-818a91325f1c400589775ec2c9eb391f71d5b095d2da4d835db883c8e8f466bc WatchSource:0}: Error finding container 818a91325f1c400589775ec2c9eb391f71d5b095d2da4d835db883c8e8f466bc: Status 404 returned error can't find the container with id 818a91325f1c400589775ec2c9eb391f71d5b095d2da4d835db883c8e8f466bc
Sep 16 19:22:43.401644 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:43.401578    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerName="main" containerID="cri-o://d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd" gracePeriod=45
Sep 16 19:22:43.618947 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:43.618909    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" event={"ID":"689b0d75-f6dc-4f06-b801-d8de1a801e73","Type":"ContainerStarted","Data":"19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1"}
Sep 16 19:22:43.619132 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:43.618962    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" event={"ID":"689b0d75-f6dc-4f06-b801-d8de1a801e73","Type":"ContainerStarted","Data":"818a91325f1c400589775ec2c9eb391f71d5b095d2da4d835db883c8e8f466bc"}
Sep 16 19:22:43.619132 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:43.618982    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:22:43.651179 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:43.651062    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" podStartSLOduration=1.65104355 podStartE2EDuration="1.65104355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:43.649510919 +0000 UTC m=+2579.173969756" watchObservedRunningTime="2026-09-16 19:22:43.65104355 +0000 UTC m=+2579.175502433"
Sep 16 19:22:52.396256 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:52.396207    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:52.396641 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:52.396352    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:22:54.034604 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:22:54.034567    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:00.013253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:00.013213    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:01.046678 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:01.046644    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:02.391040 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:02.391006    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:08.010672 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:08.010582    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:12.390981 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:12.390944    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:13.650197 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.650169    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:23:13.758196 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.758104    2590 generic.go:356] "Generic (PLEG): container finished" podID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerID="d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd" exitCode=0
Sep 16 19:23:13.758196 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.758150    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"
Sep 16 19:23:13.758405 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.758195    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" event={"ID":"7e630b67-ad9a-4b4d-8f64-638392dae017","Type":"ContainerDied","Data":"d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd"}
Sep 16 19:23:13.758405 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.758257    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992" event={"ID":"7e630b67-ad9a-4b4d-8f64-638392dae017","Type":"ContainerDied","Data":"c75433bb6c4376d024cfc1cd029d7a89b852f51f6bd26e1c4e199351210fd636"}
Sep 16 19:23:13.758405 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.758285    2590 scope.go:117] "RemoveContainer" containerID="d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd"
Sep 16 19:23:13.759166 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.759147    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e630b67-ad9a-4b4d-8f64-638392dae017-tls-certs\") pod \"7e630b67-ad9a-4b4d-8f64-638392dae017\" (UID: \"7e630b67-ad9a-4b4d-8f64-638392dae017\") "
Sep 16 19:23:13.759292 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.759213    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l559s\" (UniqueName: \"kubernetes.io/projected/7e630b67-ad9a-4b4d-8f64-638392dae017-kube-api-access-l559s\") pod \"7e630b67-ad9a-4b4d-8f64-638392dae017\" (UID: \"7e630b67-ad9a-4b4d-8f64-638392dae017\") "
Sep 16 19:23:13.761636 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.761606    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e630b67-ad9a-4b4d-8f64-638392dae017-kube-api-access-l559s" (OuterVolumeSpecName: "kube-api-access-l559s") pod "7e630b67-ad9a-4b4d-8f64-638392dae017" (UID: "7e630b67-ad9a-4b4d-8f64-638392dae017"). InnerVolumeSpecName "kube-api-access-l559s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:13.761636 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.761616    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e630b67-ad9a-4b4d-8f64-638392dae017-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e630b67-ad9a-4b4d-8f64-638392dae017" (UID: "7e630b67-ad9a-4b4d-8f64-638392dae017"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:13.776592 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.776569    2590 scope.go:117] "RemoveContainer" containerID="d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd"
Sep 16 19:23:13.776988 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:23:13.776967    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd\": container with ID starting with d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd not found: ID does not exist" containerID="d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd"
Sep 16 19:23:13.777061 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.777000    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd"} err="failed to get container status \"d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd\": rpc error: code = NotFound desc = could not find container \"d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd\": container with ID starting with d255ede83be4a2c6d7a9714002bbd5d76e2e5ad2fb8682d53f92988c22d67cbd not found: ID does not exist"
Sep 16 19:23:13.859955 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.859917    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l559s\" (UniqueName: \"kubernetes.io/projected/7e630b67-ad9a-4b4d-8f64-638392dae017-kube-api-access-l559s\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:13.859955 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:13.859950    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e630b67-ad9a-4b4d-8f64-638392dae017-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:14.081002 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:14.080966    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"]
Sep 16 19:23:14.089459 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:14.089431    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schez5992"]
Sep 16 19:23:14.625938 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:14.625904    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:23:15.026262 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:15.026230    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" path="/var/lib/kubelet/pods/7e630b67-ad9a-4b4d-8f64-638392dae017/volumes"
Sep 16 19:23:18.575473 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:18.575443    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:18.587275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:18.587241    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"]
Sep 16 19:23:18.596159 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:23:18.596123    2590 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:23:18.596159 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:23:18.596159 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:23:18.596159 ip-10-0-143-44 kubenswrapper[2590]:  > containerID="81d4303d824cae7488cf6eee98d1741866a09627feb9daf688caeb0b34ae0f5e" cmd=["/bin/sleep","15"]
Sep 16 19:23:18.596516 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:18.596181    2590 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv" message="" err=<
Sep 16 19:23:18.596516 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:23:18.596516 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:23:18.596516 ip-10-0-143-44 kubenswrapper[2590]:  >
Sep 16 19:23:18.596516 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:23:18.596198    2590 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:23:18.596516 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:23:18.596516 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:23:18.596516 ip-10-0-143-44 kubenswrapper[2590]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv" podUID="eea67604-a010-4059-b094-087449c37651" containerName="main" containerID="cri-o://81d4303d824cae7488cf6eee98d1741866a09627feb9daf688caeb0b34ae0f5e"
Sep 16 19:23:18.596516 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:18.596244    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv" podUID="eea67604-a010-4059-b094-087449c37651" containerName="main" containerID="cri-o://81d4303d824cae7488cf6eee98d1741866a09627feb9daf688caeb0b34ae0f5e" gracePeriod=60
Sep 16 19:23:18.661700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:18.661663    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:18.783377 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:18.783346    2590 generic.go:356] "Generic (PLEG): container finished" podID="eea67604-a010-4059-b094-087449c37651" containerID="81d4303d824cae7488cf6eee98d1741866a09627feb9daf688caeb0b34ae0f5e" exitCode=0
Sep 16 19:23:18.783554 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:18.783420    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv" event={"ID":"eea67604-a010-4059-b094-087449c37651","Type":"ContainerDied","Data":"81d4303d824cae7488cf6eee98d1741866a09627feb9daf688caeb0b34ae0f5e"}
Sep 16 19:23:19.168055 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.168015    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv" podUID="eea67604-a010-4059-b094-087449c37651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 16 19:23:19.348140 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.348116    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:23:19.407228 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.407197    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hszn\" (UniqueName: \"kubernetes.io/projected/eea67604-a010-4059-b094-087449c37651-kube-api-access-7hszn\") pod \"eea67604-a010-4059-b094-087449c37651\" (UID: \"eea67604-a010-4059-b094-087449c37651\") "
Sep 16 19:23:19.407430 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.407250    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea67604-a010-4059-b094-087449c37651-tls-certs\") pod \"eea67604-a010-4059-b094-087449c37651\" (UID: \"eea67604-a010-4059-b094-087449c37651\") "
Sep 16 19:23:19.407430 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.407287    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-tmp-dir\") pod \"eea67604-a010-4059-b094-087449c37651\" (UID: \"eea67604-a010-4059-b094-087449c37651\") "
Sep 16 19:23:19.407430 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.407318    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-dshm\") pod \"eea67604-a010-4059-b094-087449c37651\" (UID: \"eea67604-a010-4059-b094-087449c37651\") "
Sep 16 19:23:19.407430 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.407339    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-model-cache\") pod \"eea67604-a010-4059-b094-087449c37651\" (UID: \"eea67604-a010-4059-b094-087449c37651\") "
Sep 16 19:23:19.407430 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.407389    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-home\") pod \"eea67604-a010-4059-b094-087449c37651\" (UID: \"eea67604-a010-4059-b094-087449c37651\") "
Sep 16 19:23:19.407700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.407638    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-model-cache" (OuterVolumeSpecName: "model-cache") pod "eea67604-a010-4059-b094-087449c37651" (UID: "eea67604-a010-4059-b094-087449c37651"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:19.407700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.407644    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eea67604-a010-4059-b094-087449c37651" (UID: "eea67604-a010-4059-b094-087449c37651"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:19.407794 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.407708    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-home" (OuterVolumeSpecName: "home") pod "eea67604-a010-4059-b094-087449c37651" (UID: "eea67604-a010-4059-b094-087449c37651"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:19.409633 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.409612    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea67604-a010-4059-b094-087449c37651-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eea67604-a010-4059-b094-087449c37651" (UID: "eea67604-a010-4059-b094-087449c37651"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:19.409702 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.409642    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-dshm" (OuterVolumeSpecName: "dshm") pod "eea67604-a010-4059-b094-087449c37651" (UID: "eea67604-a010-4059-b094-087449c37651"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:19.409702 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.409692    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea67604-a010-4059-b094-087449c37651-kube-api-access-7hszn" (OuterVolumeSpecName: "kube-api-access-7hszn") pod "eea67604-a010-4059-b094-087449c37651" (UID: "eea67604-a010-4059-b094-087449c37651"). InnerVolumeSpecName "kube-api-access-7hszn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:19.508270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.508182    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:19.508270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.508211    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hszn\" (UniqueName: \"kubernetes.io/projected/eea67604-a010-4059-b094-087449c37651-kube-api-access-7hszn\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:19.508270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.508222    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea67604-a010-4059-b094-087449c37651-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:19.508270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.508231    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:19.508270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.508239    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:19.508270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.508247    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea67604-a010-4059-b094-087449c37651-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:19.789146 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.789116    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"
Sep 16 19:23:19.789639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.789119    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv" event={"ID":"eea67604-a010-4059-b094-087449c37651","Type":"ContainerDied","Data":"13a6a1ccfd778718860d993a1a2d499d0297cda2aa6d61b79267044a8a33e263"}
Sep 16 19:23:19.789639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.789273    2590 scope.go:117] "RemoveContainer" containerID="81d4303d824cae7488cf6eee98d1741866a09627feb9daf688caeb0b34ae0f5e"
Sep 16 19:23:19.811757 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.811731    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"]
Sep 16 19:23:19.823313 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:19.823279    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-6gdrv"]
Sep 16 19:23:21.025342 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:21.025304    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea67604-a010-4059-b094-087449c37651" path="/var/lib/kubelet/pods/eea67604-a010-4059-b094-087449c37651/volumes"
Sep 16 19:23:21.551046 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:21.550998    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:24.289730 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:24.289699    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:24.393406 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:24.393377    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:47.932276 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:47.932241    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:48.031282 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:23:48.031249    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:03.406557 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:03.406500    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:08.711869 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:08.711832    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:08.806423 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:08.806384    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:08.909410 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:08.909377    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:16.263457 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:16.263422    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:16.643137 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:16.643104    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:20.570714 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.570680    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"]
Sep 16 19:24:20.571094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.570981    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerName="main"
Sep 16 19:24:20.571094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.570993    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerName="main"
Sep 16 19:24:20.571094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.571003    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea67604-a010-4059-b094-087449c37651" containerName="main"
Sep 16 19:24:20.571094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.571008    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea67604-a010-4059-b094-087449c37651" containerName="main"
Sep 16 19:24:20.571094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.571057    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e630b67-ad9a-4b4d-8f64-638392dae017" containerName="main"
Sep 16 19:24:20.571094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.571067    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea67604-a010-4059-b094-087449c37651" containerName="main"
Sep 16 19:24:20.573517 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.573500    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.576249 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.576226    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-jmlm9\""
Sep 16 19:24:20.577022 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.577000    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:24:20.577129 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.577030    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:24:20.577129 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.577000    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:24:20.591250 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.591224    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"]
Sep 16 19:24:20.603139 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.603111    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:20.728465 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.728432    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.728465 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.728468    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.728701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.728494    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.728701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.728554    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.728701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.728604    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.728701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.728625    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtpg\" (UniqueName: \"kubernetes.io/projected/e0cfbe48-9ceb-4928-933c-e8f619bd917c-kube-api-access-hhtpg\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.829797 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.829698    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtpg\" (UniqueName: \"kubernetes.io/projected/e0cfbe48-9ceb-4928-933c-e8f619bd917c-kube-api-access-hhtpg\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.829797 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.829742    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.830072 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.829872    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.830072 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.829960    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.830072 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.830008    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.830217 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.830081    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.830217 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.830149    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.830294 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.830241    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.830372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.830357    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.832306 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.832284    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.832688 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.832668    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.847213 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.847182    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:20.855018 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.854997    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtpg\" (UniqueName: \"kubernetes.io/projected/e0cfbe48-9ceb-4928-933c-e8f619bd917c-kube-api-access-hhtpg\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:20.883849 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:20.883814    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:21.168661 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:21.168636    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"]
Sep 16 19:24:21.170499 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:24:21.170473    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cfbe48_9ceb_4928_933c_e8f619bd917c.slice/crio-d13304f8c5264ce290ef10802f1b5ae46ede86700592e9b5cbb94684e25b16bc WatchSource:0}: Error finding container d13304f8c5264ce290ef10802f1b5ae46ede86700592e9b5cbb94684e25b16bc: Status 404 returned error can't find the container with id d13304f8c5264ce290ef10802f1b5ae46ede86700592e9b5cbb94684e25b16bc
Sep 16 19:24:22.062505 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:22.062464    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk" event={"ID":"e0cfbe48-9ceb-4928-933c-e8f619bd917c","Type":"ContainerStarted","Data":"2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a"}
Sep 16 19:24:22.062944 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:22.062553    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk" event={"ID":"e0cfbe48-9ceb-4928-933c-e8f619bd917c","Type":"ContainerStarted","Data":"d13304f8c5264ce290ef10802f1b5ae46ede86700592e9b5cbb94684e25b16bc"}
Sep 16 19:24:22.094418 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:22.094362    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk" podStartSLOduration=2.094345051 podStartE2EDuration="2.094345051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:22.093488557 +0000 UTC m=+2677.617947402" watchObservedRunningTime="2026-09-16 19:24:22.094345051 +0000 UTC m=+2677.618803885"
Sep 16 19:24:22.884440 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:22.884406    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:24.342187 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:24.342154    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:24.446205 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:24.446171    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:30.884674 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:30.884572    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:30.902757 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:30.902729    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:31.111705 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:31.111666    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:24:33.560748 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:33.560713    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:47.636706 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:47.636672    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:47.665189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:47.665153    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"]
Sep 16 19:24:47.734389 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:47.734352    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:52.037510 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:52.037462    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:54.626145 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:24:54.626098    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" podUID="689b0d75-f6dc-4f06-b801-d8de1a801e73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:00.967412 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:00.967379    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:01.220785 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.220702    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"]
Sep 16 19:25:01.223990 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.223966    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:01.227234 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.227208    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-dwlw7\""
Sep 16 19:25:01.227384 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.227284    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:25:01.238480 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.238449    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"]
Sep 16 19:25:01.263020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.262986    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:01.266126 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.266100    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7n7\" (UniqueName: \"kubernetes.io/projected/62d2b7a7-91cc-4822-b71e-e398571877df-kube-api-access-pg7n7\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq\" (UID: \"62d2b7a7-91cc-4822-b71e-e398571877df\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:01.266213 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.266199    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d2b7a7-91cc-4822-b71e-e398571877df-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq\" (UID: \"62d2b7a7-91cc-4822-b71e-e398571877df\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:01.366841 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.366803    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d2b7a7-91cc-4822-b71e-e398571877df-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq\" (UID: \"62d2b7a7-91cc-4822-b71e-e398571877df\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:01.366841 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.366855    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7n7\" (UniqueName: \"kubernetes.io/projected/62d2b7a7-91cc-4822-b71e-e398571877df-kube-api-access-pg7n7\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq\" (UID: \"62d2b7a7-91cc-4822-b71e-e398571877df\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:01.369518 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.369498    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d2b7a7-91cc-4822-b71e-e398571877df-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq\" (UID: \"62d2b7a7-91cc-4822-b71e-e398571877df\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:01.376019 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.375991    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7n7\" (UniqueName: \"kubernetes.io/projected/62d2b7a7-91cc-4822-b71e-e398571877df-kube-api-access-pg7n7\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq\" (UID: \"62d2b7a7-91cc-4822-b71e-e398571877df\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:01.536912 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.536854    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:01.820337 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:01.820241    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"]
Sep 16 19:25:01.823767 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:25:01.823738    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d2b7a7_91cc_4822_b71e_e398571877df.slice/crio-b4151a15c670290311f26506fbe25d084d88620cb9a7888c2f6bb9b440519b91 WatchSource:0}: Error finding container b4151a15c670290311f26506fbe25d084d88620cb9a7888c2f6bb9b440519b91: Status 404 returned error can't find the container with id b4151a15c670290311f26506fbe25d084d88620cb9a7888c2f6bb9b440519b91
Sep 16 19:25:02.234611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:02.234512    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" event={"ID":"62d2b7a7-91cc-4822-b71e-e398571877df","Type":"ContainerStarted","Data":"20b1fc3919138d7fa6912577be13692cfec62ef2d7947748cd5774a9ea0fe570"}
Sep 16 19:25:02.234611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:02.234564    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" event={"ID":"62d2b7a7-91cc-4822-b71e-e398571877df","Type":"ContainerStarted","Data":"b4151a15c670290311f26506fbe25d084d88620cb9a7888c2f6bb9b440519b91"}
Sep 16 19:25:02.234611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:02.234593    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:02.266489 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:02.266411    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" podStartSLOduration=1.2663884 podStartE2EDuration="1.2663884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:02.264657341 +0000 UTC m=+2717.789116175" watchObservedRunningTime="2026-09-16 19:25:02.2663884 +0000 UTC m=+2717.790847233"
Sep 16 19:25:02.684640 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:02.684604    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" podUID="689b0d75-f6dc-4f06-b801-d8de1a801e73" containerName="main" containerID="cri-o://19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1" gracePeriod=45
Sep 16 19:25:04.626243 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:04.626209    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" podUID="689b0d75-f6dc-4f06-b801-d8de1a801e73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:14.552141 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:14.552102    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:14.626151 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:14.626118    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" podUID="689b0d75-f6dc-4f06-b801-d8de1a801e73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:14.626310 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:14.626226    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:25:24.626560 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:24.626514    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" podUID="689b0d75-f6dc-4f06-b801-d8de1a801e73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:32.937832 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:32.937803    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:25:33.130411 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.130373    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/689b0d75-f6dc-4f06-b801-d8de1a801e73-tls-certs\") pod \"689b0d75-f6dc-4f06-b801-d8de1a801e73\" (UID: \"689b0d75-f6dc-4f06-b801-d8de1a801e73\") "
Sep 16 19:25:33.130611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.130470    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbj2j\" (UniqueName: \"kubernetes.io/projected/689b0d75-f6dc-4f06-b801-d8de1a801e73-kube-api-access-fbj2j\") pod \"689b0d75-f6dc-4f06-b801-d8de1a801e73\" (UID: \"689b0d75-f6dc-4f06-b801-d8de1a801e73\") "
Sep 16 19:25:33.132967 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.132939    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/689b0d75-f6dc-4f06-b801-d8de1a801e73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "689b0d75-f6dc-4f06-b801-d8de1a801e73" (UID: "689b0d75-f6dc-4f06-b801-d8de1a801e73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:33.133095 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.132969    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689b0d75-f6dc-4f06-b801-d8de1a801e73-kube-api-access-fbj2j" (OuterVolumeSpecName: "kube-api-access-fbj2j") pod "689b0d75-f6dc-4f06-b801-d8de1a801e73" (UID: "689b0d75-f6dc-4f06-b801-d8de1a801e73"). InnerVolumeSpecName "kube-api-access-fbj2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:33.232192 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.232154    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbj2j\" (UniqueName: \"kubernetes.io/projected/689b0d75-f6dc-4f06-b801-d8de1a801e73-kube-api-access-fbj2j\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:25:33.232192 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.232187    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/689b0d75-f6dc-4f06-b801-d8de1a801e73-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:25:33.241383 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.241358    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:25:33.375530 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.375499    2590 generic.go:356] "Generic (PLEG): container finished" podID="689b0d75-f6dc-4f06-b801-d8de1a801e73" containerID="19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1" exitCode=0
Sep 16 19:25:33.375700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.375548    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"
Sep 16 19:25:33.375700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.375551    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" event={"ID":"689b0d75-f6dc-4f06-b801-d8de1a801e73","Type":"ContainerDied","Data":"19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1"}
Sep 16 19:25:33.375700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.375679    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd" event={"ID":"689b0d75-f6dc-4f06-b801-d8de1a801e73","Type":"ContainerDied","Data":"818a91325f1c400589775ec2c9eb391f71d5b095d2da4d835db883c8e8f466bc"}
Sep 16 19:25:33.375826 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.375708    2590 scope.go:117] "RemoveContainer" containerID="19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1"
Sep 16 19:25:33.384845 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.384828    2590 scope.go:117] "RemoveContainer" containerID="19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1"
Sep 16 19:25:33.385146 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:25:33.385130    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1\": container with ID starting with 19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1 not found: ID does not exist" containerID="19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1"
Sep 16 19:25:33.385218 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.385153    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1"} err="failed to get container status \"19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1\": rpc error: code = NotFound desc = could not find container \"19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1\": container with ID starting with 19aa39486885ff26380da2ced61bb0a16a6f734a560e24cf453ba368ba010af1 not found: ID does not exist"
Sep 16 19:25:33.396345 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.396310    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"]
Sep 16 19:25:33.398049 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:33.398024    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherdnsd"]
Sep 16 19:25:34.430382 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:34.430349    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"]
Sep 16 19:25:34.431153 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:34.431123    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:34.498752 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:34.498722    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:35.025355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:35.025324    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="689b0d75-f6dc-4f06-b801-d8de1a801e73" path="/var/lib/kubelet/pods/689b0d75-f6dc-4f06-b801-d8de1a801e73/volumes"
Sep 16 19:25:37.551271 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:37.551238    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:37.646518 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:37.646488    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:40.940591 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:40.940552    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:41.029293 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:41.029264    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:43.242036 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:43.241997    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" podUID="62d2b7a7-91cc-4822-b71e-e398571877df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:44.252467 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:44.252436    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:49.451350 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:49.451289    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" podUID="62d2b7a7-91cc-4822-b71e-e398571877df" containerName="main" containerID="cri-o://20b1fc3919138d7fa6912577be13692cfec62ef2d7947748cd5774a9ea0fe570" gracePeriod=45
Sep 16 19:25:50.196055 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:50.196027    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:50.297560 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:50.297530    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:53.241674 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:25:53.241638    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" podUID="62d2b7a7-91cc-4822-b71e-e398571877df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:01.516998 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:01.516959    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:03.241081 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:03.241036    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" podUID="62d2b7a7-91cc-4822-b71e-e398571877df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:03.241565 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:03.241173    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:26:13.241263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:13.241169    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" podUID="62d2b7a7-91cc-4822-b71e-e398571877df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:19.578579 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:19.578547    2590 generic.go:356] "Generic (PLEG): container finished" podID="62d2b7a7-91cc-4822-b71e-e398571877df" containerID="20b1fc3919138d7fa6912577be13692cfec62ef2d7947748cd5774a9ea0fe570" exitCode=0
Sep 16 19:26:19.578947 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:19.578616    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" event={"ID":"62d2b7a7-91cc-4822-b71e-e398571877df","Type":"ContainerDied","Data":"20b1fc3919138d7fa6912577be13692cfec62ef2d7947748cd5774a9ea0fe570"}
Sep 16 19:26:19.702133 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:19.702106    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:26:19.816591 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:19.816551    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg7n7\" (UniqueName: \"kubernetes.io/projected/62d2b7a7-91cc-4822-b71e-e398571877df-kube-api-access-pg7n7\") pod \"62d2b7a7-91cc-4822-b71e-e398571877df\" (UID: \"62d2b7a7-91cc-4822-b71e-e398571877df\") "
Sep 16 19:26:19.816800 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:19.816645    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d2b7a7-91cc-4822-b71e-e398571877df-tls-certs\") pod \"62d2b7a7-91cc-4822-b71e-e398571877df\" (UID: \"62d2b7a7-91cc-4822-b71e-e398571877df\") "
Sep 16 19:26:19.819056 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:19.819025    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d2b7a7-91cc-4822-b71e-e398571877df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62d2b7a7-91cc-4822-b71e-e398571877df" (UID: "62d2b7a7-91cc-4822-b71e-e398571877df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:19.819157 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:19.819082    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d2b7a7-91cc-4822-b71e-e398571877df-kube-api-access-pg7n7" (OuterVolumeSpecName: "kube-api-access-pg7n7") pod "62d2b7a7-91cc-4822-b71e-e398571877df" (UID: "62d2b7a7-91cc-4822-b71e-e398571877df"). InnerVolumeSpecName "kube-api-access-pg7n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:19.917534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:19.917494    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d2b7a7-91cc-4822-b71e-e398571877df-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:19.917534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:19.917526    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg7n7\" (UniqueName: \"kubernetes.io/projected/62d2b7a7-91cc-4822-b71e-e398571877df-kube-api-access-pg7n7\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:20.584483 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:20.584445    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"
Sep 16 19:26:20.584483 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:20.584468    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq" event={"ID":"62d2b7a7-91cc-4822-b71e-e398571877df","Type":"ContainerDied","Data":"b4151a15c670290311f26506fbe25d084d88620cb9a7888c2f6bb9b440519b91"}
Sep 16 19:26:20.585033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:20.584535    2590 scope.go:117] "RemoveContainer" containerID="20b1fc3919138d7fa6912577be13692cfec62ef2d7947748cd5774a9ea0fe570"
Sep 16 19:26:20.605513 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:20.605479    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"]
Sep 16 19:26:20.609611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:20.609584    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezcvbq"]
Sep 16 19:26:21.025572 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:21.025546    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d2b7a7-91cc-4822-b71e-e398571877df" path="/var/lib/kubelet/pods/62d2b7a7-91cc-4822-b71e-e398571877df/volumes"
Sep 16 19:26:24.679936 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:24.679899    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"]
Sep 16 19:26:24.689061 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:26:24.689023    2590 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:26:24.689061 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:26:24.689061 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:26:24.689061 ip-10-0-143-44 kubenswrapper[2590]:  > containerID="2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a" cmd=["/bin/sleep","15"]
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:24.689074    2590 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk" message="" err=<
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]:  >
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:26:24.689090    2590 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk" podUID="e0cfbe48-9ceb-4928-933c-e8f619bd917c" containerName="main" containerID="cri-o://2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a"
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:24.689133    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk" podUID="e0cfbe48-9ceb-4928-933c-e8f619bd917c" containerName="main" containerID="cri-o://2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a" gracePeriod=60
Sep 16 19:26:24.689318 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:24.689144    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:24.750199 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:24.750159    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:25.033138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.033108    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:26:25.165347 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.165316    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tls-certs\") pod \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") "
Sep 16 19:26:25.165544 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.165363    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhtpg\" (UniqueName: \"kubernetes.io/projected/e0cfbe48-9ceb-4928-933c-e8f619bd917c-kube-api-access-hhtpg\") pod \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") "
Sep 16 19:26:25.165544 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.165384    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-dshm\") pod \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") "
Sep 16 19:26:25.165544 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.165408    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-home\") pod \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") "
Sep 16 19:26:25.165544 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.165439    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-model-cache\") pod \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") "
Sep 16 19:26:25.165544 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.165483    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tmp-dir\") pod \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\" (UID: \"e0cfbe48-9ceb-4928-933c-e8f619bd917c\") "
Sep 16 19:26:25.165820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.165801    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-home" (OuterVolumeSpecName: "home") pod "e0cfbe48-9ceb-4928-933c-e8f619bd917c" (UID: "e0cfbe48-9ceb-4928-933c-e8f619bd917c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:25.165894 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.165816    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0cfbe48-9ceb-4928-933c-e8f619bd917c" (UID: "e0cfbe48-9ceb-4928-933c-e8f619bd917c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:25.165894 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.165825    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0cfbe48-9ceb-4928-933c-e8f619bd917c" (UID: "e0cfbe48-9ceb-4928-933c-e8f619bd917c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:25.167741 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.167714    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-dshm" (OuterVolumeSpecName: "dshm") pod "e0cfbe48-9ceb-4928-933c-e8f619bd917c" (UID: "e0cfbe48-9ceb-4928-933c-e8f619bd917c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:25.167844 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.167782    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cfbe48-9ceb-4928-933c-e8f619bd917c-kube-api-access-hhtpg" (OuterVolumeSpecName: "kube-api-access-hhtpg") pod "e0cfbe48-9ceb-4928-933c-e8f619bd917c" (UID: "e0cfbe48-9ceb-4928-933c-e8f619bd917c"). InnerVolumeSpecName "kube-api-access-hhtpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:25.168086 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.168070    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0cfbe48-9ceb-4928-933c-e8f619bd917c" (UID: "e0cfbe48-9ceb-4928-933c-e8f619bd917c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:25.266892 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.266788    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:25.266892 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.266818    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0cfbe48-9ceb-4928-933c-e8f619bd917c-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:25.266892 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.266828    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhtpg\" (UniqueName: \"kubernetes.io/projected/e0cfbe48-9ceb-4928-933c-e8f619bd917c-kube-api-access-hhtpg\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:25.266892 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.266836    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:25.266892 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.266846    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:25.266892 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.266853    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0cfbe48-9ceb-4928-933c-e8f619bd917c-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:25.608843 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.608809    2590 generic.go:356] "Generic (PLEG): container finished" podID="e0cfbe48-9ceb-4928-933c-e8f619bd917c" containerID="2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a" exitCode=0
Sep 16 19:26:25.609043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.608851    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"
Sep 16 19:26:25.609043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.608897    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk" event={"ID":"e0cfbe48-9ceb-4928-933c-e8f619bd917c","Type":"ContainerDied","Data":"2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a"}
Sep 16 19:26:25.609043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.608943    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk" event={"ID":"e0cfbe48-9ceb-4928-933c-e8f619bd917c","Type":"ContainerDied","Data":"d13304f8c5264ce290ef10802f1b5ae46ede86700592e9b5cbb94684e25b16bc"}
Sep 16 19:26:25.609043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.608967    2590 scope.go:117] "RemoveContainer" containerID="2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a"
Sep 16 19:26:25.617841 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.617815    2590 scope.go:117] "RemoveContainer" containerID="2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a"
Sep 16 19:26:25.618210 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:26:25.618188    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a\": container with ID starting with 2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a not found: ID does not exist" containerID="2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a"
Sep 16 19:26:25.618263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.618218    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a"} err="failed to get container status \"2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a\": rpc error: code = NotFound desc = could not find container \"2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a\": container with ID starting with 2e6f002700f032023139cbabaf03cb6c916ed39114c32981a2a767e41813d05a not found: ID does not exist"
Sep 16 19:26:25.629603 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.629570    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"]
Sep 16 19:26:25.635385 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:25.635356    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969f84pk"]
Sep 16 19:26:27.031162 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:26:27.031124    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0cfbe48-9ceb-4928-933c-e8f619bd917c" path="/var/lib/kubelet/pods/e0cfbe48-9ceb-4928-933c-e8f619bd917c/volumes"
Sep 16 19:27:14.756786 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:14.756749    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:16.186402 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:16.186360    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:22.310153 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:22.310114    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:23.251060 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251028    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"]
Sep 16 19:27:23.251340 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251328    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689b0d75-f6dc-4f06-b801-d8de1a801e73" containerName="main"
Sep 16 19:27:23.251385 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251342    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="689b0d75-f6dc-4f06-b801-d8de1a801e73" containerName="main"
Sep 16 19:27:23.251385 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251357    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cfbe48-9ceb-4928-933c-e8f619bd917c" containerName="main"
Sep 16 19:27:23.251385 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251363    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cfbe48-9ceb-4928-933c-e8f619bd917c" containerName="main"
Sep 16 19:27:23.251385 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251377    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d2b7a7-91cc-4822-b71e-e398571877df" containerName="main"
Sep 16 19:27:23.251385 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251383    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d2b7a7-91cc-4822-b71e-e398571877df" containerName="main"
Sep 16 19:27:23.251545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251427    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cfbe48-9ceb-4928-933c-e8f619bd917c" containerName="main"
Sep 16 19:27:23.251545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251435    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d2b7a7-91cc-4822-b71e-e398571877df" containerName="main"
Sep 16 19:27:23.251545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.251441    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="689b0d75-f6dc-4f06-b801-d8de1a801e73" containerName="main"
Sep 16 19:27:23.254306 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.254287    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.256515 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.256492    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 19:27:23.256636 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.256586    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-w9tf2\""
Sep 16 19:27:23.256703 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.256658    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:27:23.257009 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.256989    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:27:23.257093 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.257048    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:27:23.257159 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.257139    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:27:23.269203 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.269178    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"]
Sep 16 19:27:23.292195 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.292164    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:23.326280 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.326245    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgs6h\" (UniqueName: \"kubernetes.io/projected/905e0db1-e120-45f5-8ddc-54685e7a50dc-kube-api-access-rgs6h\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.326639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.326296    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.326639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.326316    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905e0db1-e120-45f5-8ddc-54685e7a50dc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.326639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.326420    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/905e0db1-e120-45f5-8ddc-54685e7a50dc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.326639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.326454    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.326639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.326488    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.326639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.326519    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.326639 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.326551    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.427481 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.427447    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.427695 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.427489    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.427695 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.427519    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.427695 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.427542    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgs6h\" (UniqueName: \"kubernetes.io/projected/905e0db1-e120-45f5-8ddc-54685e7a50dc-kube-api-access-rgs6h\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.427695 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.427576    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.427695 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.427606    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905e0db1-e120-45f5-8ddc-54685e7a50dc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.427695 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.427638    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/905e0db1-e120-45f5-8ddc-54685e7a50dc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.427695 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.427665    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.428110 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.427993    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.428110 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.428023    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.428389 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.428357    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/905e0db1-e120-45f5-8ddc-54685e7a50dc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.428502 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.428489    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.428575 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.428513    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.429992 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.429970    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.430248 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.430228    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905e0db1-e120-45f5-8ddc-54685e7a50dc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.437350 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.437326    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgs6h\" (UniqueName: \"kubernetes.io/projected/905e0db1-e120-45f5-8ddc-54685e7a50dc-kube-api-access-rgs6h\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.513725 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.513642    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"]
Sep 16 19:27:23.517504 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.517482    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:27:23.520392 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.520372    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-4cksl\""
Sep 16 19:27:23.530699 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.530676    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"]
Sep 16 19:27:23.554003 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.553970    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:23.564777 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.564754    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:23.629600 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.629523    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d361e-76fd-4e2e-9626-7b6796c7fea2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8\" (UID: \"060d361e-76fd-4e2e-9626-7b6796c7fea2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:27:23.629950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.629870    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnb7\" (UniqueName: \"kubernetes.io/projected/060d361e-76fd-4e2e-9626-7b6796c7fea2-kube-api-access-djnb7\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8\" (UID: \"060d361e-76fd-4e2e-9626-7b6796c7fea2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:27:23.731235 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.731197    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d361e-76fd-4e2e-9626-7b6796c7fea2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8\" (UID: \"060d361e-76fd-4e2e-9626-7b6796c7fea2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:27:23.731391 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.731242    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnb7\" (UniqueName: \"kubernetes.io/projected/060d361e-76fd-4e2e-9626-7b6796c7fea2-kube-api-access-djnb7\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8\" (UID: \"060d361e-76fd-4e2e-9626-7b6796c7fea2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:27:23.734008 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.733980    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d361e-76fd-4e2e-9626-7b6796c7fea2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8\" (UID: \"060d361e-76fd-4e2e-9626-7b6796c7fea2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:27:23.745290 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.745258    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnb7\" (UniqueName: \"kubernetes.io/projected/060d361e-76fd-4e2e-9626-7b6796c7fea2-kube-api-access-djnb7\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8\" (UID: \"060d361e-76fd-4e2e-9626-7b6796c7fea2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:27:23.827376 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.827346    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:27:23.861905 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:23.861861    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"]
Sep 16 19:27:23.864462 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:27:23.864428    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905e0db1_e120_45f5_8ddc_54685e7a50dc.slice/crio-35415cb76dc5dca0306e5139dc092f14c88b0ed2479a0f67f6a224d889fc57e4 WatchSource:0}: Error finding container 35415cb76dc5dca0306e5139dc092f14c88b0ed2479a0f67f6a224d889fc57e4: Status 404 returned error can't find the container with id 35415cb76dc5dca0306e5139dc092f14c88b0ed2479a0f67f6a224d889fc57e4
Sep 16 19:27:24.104452 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:24.104423    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"]
Sep 16 19:27:24.107393 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:27:24.107362    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060d361e_76fd_4e2e_9626_7b6796c7fea2.slice/crio-d37d6383aab9bda47453ce009e46cb59734b28b801779b6ce8ef8eafe12254c9 WatchSource:0}: Error finding container d37d6383aab9bda47453ce009e46cb59734b28b801779b6ce8ef8eafe12254c9: Status 404 returned error can't find the container with id d37d6383aab9bda47453ce009e46cb59734b28b801779b6ce8ef8eafe12254c9
Sep 16 19:27:24.866585 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:24.866544    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" event={"ID":"060d361e-76fd-4e2e-9626-7b6796c7fea2","Type":"ContainerStarted","Data":"36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df"}
Sep 16 19:27:24.867086 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:24.866704    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:27:24.867086 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:24.866740    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" event={"ID":"060d361e-76fd-4e2e-9626-7b6796c7fea2","Type":"ContainerStarted","Data":"d37d6383aab9bda47453ce009e46cb59734b28b801779b6ce8ef8eafe12254c9"}
Sep 16 19:27:24.868237 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:24.868209    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" event={"ID":"905e0db1-e120-45f5-8ddc-54685e7a50dc","Type":"ContainerStarted","Data":"3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc"}
Sep 16 19:27:24.868372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:24.868246    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" event={"ID":"905e0db1-e120-45f5-8ddc-54685e7a50dc","Type":"ContainerStarted","Data":"35415cb76dc5dca0306e5139dc092f14c88b0ed2479a0f67f6a224d889fc57e4"}
Sep 16 19:27:24.886137 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:24.886081    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" podStartSLOduration=1.886062956 podStartE2EDuration="1.886062956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:24.88544531 +0000 UTC m=+2860.409904148" watchObservedRunningTime="2026-09-16 19:27:24.886062956 +0000 UTC m=+2860.410521791"
Sep 16 19:27:27.885562 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:27.885520    2590 generic.go:356] "Generic (PLEG): container finished" podID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerID="3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc" exitCode=0
Sep 16 19:27:27.885974 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:27.885580    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" event={"ID":"905e0db1-e120-45f5-8ddc-54685e7a50dc","Type":"ContainerDied","Data":"3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc"}
Sep 16 19:27:28.479744 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:28.479704    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:28.548145 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:28.548102    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:28.892169 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:28.892129    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" event={"ID":"905e0db1-e120-45f5-8ddc-54685e7a50dc","Type":"ContainerStarted","Data":"85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601"}
Sep 16 19:27:28.922937 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:28.922860    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podStartSLOduration=2.945124841 podStartE2EDuration="5.922835949s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.977711108s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:28.920477338 +0000 UTC m=+2864.444936174" watchObservedRunningTime="2026-09-16 19:27:28.922835949 +0000 UTC m=+2864.447294784"
Sep 16 19:27:29.565468 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:29.565430    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:31.990712 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:31.990674    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:33.565188 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:33.565140    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:27:33.566749 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:33.566720    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:27:43.565267 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:43.565223    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:27:53.565813 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:53.565750    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:27:55.874940 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:27:55.874904    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:28:03.565251 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:28:03.565212    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:28:13.566157 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:28:13.566100    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:28:15.918373 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:28:15.918339    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:15.980062 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:28:15.980023    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:23.565693 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:28:23.565633    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:28:33.565530 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:28:33.565488    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:28:43.575409 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:28:43.575372    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:28:43.584130 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:28:43.584100    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:29:06.048111 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:06.048013    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:06.142674 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:06.142638    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:12.777543 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:12.777507    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:13.162850 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:13.162761    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:13.440697 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:13.440607    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:15.997594 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:15.997555    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"]
Sep 16 19:29:16.001944 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:16.001913    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"]
Sep 16 19:29:16.011654 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:16.011619    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:16.070662 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:16.070629    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:18.757899 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:18.757850    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:18.826330 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:18.826298    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:22.914809 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:22.914769    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:25.875313 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:25.875275    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" podUID="060d361e-76fd-4e2e-9626-7b6796c7fea2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:27.884606 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:27.884572    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:31.022725 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.022671    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main" containerID="cri-o://85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601" gracePeriod=45
Sep 16 19:29:31.027270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.027234    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" podUID="060d361e-76fd-4e2e-9626-7b6796c7fea2" containerName="main" containerID="cri-o://36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df" gracePeriod=45
Sep 16 19:29:31.576123 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.576081    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:29:31.670329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.670304    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:29:31.734281 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734243    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-kserve-provision-location\") pod \"905e0db1-e120-45f5-8ddc-54685e7a50dc\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") "
Sep 16 19:29:31.734281 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734284    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgs6h\" (UniqueName: \"kubernetes.io/projected/905e0db1-e120-45f5-8ddc-54685e7a50dc-kube-api-access-rgs6h\") pod \"905e0db1-e120-45f5-8ddc-54685e7a50dc\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") "
Sep 16 19:29:31.734556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734315    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-dshm\") pod \"905e0db1-e120-45f5-8ddc-54685e7a50dc\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") "
Sep 16 19:29:31.734556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734341    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/905e0db1-e120-45f5-8ddc-54685e7a50dc-cabundle-cert\") pod \"905e0db1-e120-45f5-8ddc-54685e7a50dc\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") "
Sep 16 19:29:31.734556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734370    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-tmp-dir\") pod \"905e0db1-e120-45f5-8ddc-54685e7a50dc\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") "
Sep 16 19:29:31.734556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734389    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-home\") pod \"905e0db1-e120-45f5-8ddc-54685e7a50dc\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") "
Sep 16 19:29:31.734556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734409    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905e0db1-e120-45f5-8ddc-54685e7a50dc-tls-certs\") pod \"905e0db1-e120-45f5-8ddc-54685e7a50dc\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") "
Sep 16 19:29:31.734814 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734743    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905e0db1-e120-45f5-8ddc-54685e7a50dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "905e0db1-e120-45f5-8ddc-54685e7a50dc" (UID: "905e0db1-e120-45f5-8ddc-54685e7a50dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:31.734924 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734872    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-model-cache\") pod \"905e0db1-e120-45f5-8ddc-54685e7a50dc\" (UID: \"905e0db1-e120-45f5-8ddc-54685e7a50dc\") "
Sep 16 19:29:31.735402 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.734996    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-home" (OuterVolumeSpecName: "home") pod "905e0db1-e120-45f5-8ddc-54685e7a50dc" (UID: "905e0db1-e120-45f5-8ddc-54685e7a50dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:31.735402 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.735245    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "905e0db1-e120-45f5-8ddc-54685e7a50dc" (UID: "905e0db1-e120-45f5-8ddc-54685e7a50dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:31.735402 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.735339    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/905e0db1-e120-45f5-8ddc-54685e7a50dc-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:29:31.735402 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.735360    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:29:31.735402 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.735373    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:29:31.737275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.737251    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/905e0db1-e120-45f5-8ddc-54685e7a50dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "905e0db1-e120-45f5-8ddc-54685e7a50dc" (UID: "905e0db1-e120-45f5-8ddc-54685e7a50dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:31.737440 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.737417    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905e0db1-e120-45f5-8ddc-54685e7a50dc-kube-api-access-rgs6h" (OuterVolumeSpecName: "kube-api-access-rgs6h") pod "905e0db1-e120-45f5-8ddc-54685e7a50dc" (UID: "905e0db1-e120-45f5-8ddc-54685e7a50dc"). InnerVolumeSpecName "kube-api-access-rgs6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:31.737440 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.737420    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-dshm" (OuterVolumeSpecName: "dshm") pod "905e0db1-e120-45f5-8ddc-54685e7a50dc" (UID: "905e0db1-e120-45f5-8ddc-54685e7a50dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:31.746858 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.746827    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "905e0db1-e120-45f5-8ddc-54685e7a50dc" (UID: "905e0db1-e120-45f5-8ddc-54685e7a50dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:31.761294 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.761265    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "905e0db1-e120-45f5-8ddc-54685e7a50dc" (UID: "905e0db1-e120-45f5-8ddc-54685e7a50dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:31.836211 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.836170    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:29:31.836211 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.836199    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:29:31.836211 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.836207    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905e0db1-e120-45f5-8ddc-54685e7a50dc-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:29:31.836211 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.836218    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905e0db1-e120-45f5-8ddc-54685e7a50dc-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:29:31.836486 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:31.836229    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgs6h\" (UniqueName: \"kubernetes.io/projected/905e0db1-e120-45f5-8ddc-54685e7a50dc-kube-api-access-rgs6h\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:29:32.470061 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.470017    2590 generic.go:356] "Generic (PLEG): container finished" podID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerID="85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601" exitCode=0
Sep 16 19:29:32.470460 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.470075    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"
Sep 16 19:29:32.470460 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.470095    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" event={"ID":"905e0db1-e120-45f5-8ddc-54685e7a50dc","Type":"ContainerDied","Data":"85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601"}
Sep 16 19:29:32.470460 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.470152    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg" event={"ID":"905e0db1-e120-45f5-8ddc-54685e7a50dc","Type":"ContainerDied","Data":"35415cb76dc5dca0306e5139dc092f14c88b0ed2479a0f67f6a224d889fc57e4"}
Sep 16 19:29:32.470460 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.470175    2590 scope.go:117] "RemoveContainer" containerID="85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601"
Sep 16 19:29:32.479239 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.479222    2590 scope.go:117] "RemoveContainer" containerID="3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc"
Sep 16 19:29:32.487243 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.487223    2590 scope.go:117] "RemoveContainer" containerID="85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601"
Sep 16 19:29:32.487533 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:29:32.487511    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601\": container with ID starting with 85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601 not found: ID does not exist" containerID="85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601"
Sep 16 19:29:32.487588 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.487542    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601"} err="failed to get container status \"85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601\": rpc error: code = NotFound desc = could not find container \"85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601\": container with ID starting with 85993e93ba27e4d098df8d94a52e1cc96b77ad5861fcd11f98a3910d58b22601 not found: ID does not exist"
Sep 16 19:29:32.487588 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.487559    2590 scope.go:117] "RemoveContainer" containerID="3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc"
Sep 16 19:29:32.487790 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:29:32.487767    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc\": container with ID starting with 3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc not found: ID does not exist" containerID="3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc"
Sep 16 19:29:32.487848 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.487799    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc"} err="failed to get container status \"3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc\": rpc error: code = NotFound desc = could not find container \"3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc\": container with ID starting with 3a1ee7efa931314383e0d601d9010f6ad434f565ec8ba1ec73f2e554cabe9fdc not found: ID does not exist"
Sep 16 19:29:32.492381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.492360    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"]
Sep 16 19:29:32.495637 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:32.495614    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-667b89d987-dk7mg"]
Sep 16 19:29:33.026461 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:33.026428    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" path="/var/lib/kubelet/pods/905e0db1-e120-45f5-8ddc-54685e7a50dc/volumes"
Sep 16 19:29:35.875065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:35.875029    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" podUID="060d361e-76fd-4e2e-9626-7b6796c7fea2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:45.875503 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:45.875461    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" podUID="060d361e-76fd-4e2e-9626-7b6796c7fea2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:45.875926 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:45.875568    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:29:55.875353 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:29:55.875317    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" podUID="060d361e-76fd-4e2e-9626-7b6796c7fea2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:01.271281 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.271244    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:30:01.374893 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.374782    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d361e-76fd-4e2e-9626-7b6796c7fea2-tls-certs\") pod \"060d361e-76fd-4e2e-9626-7b6796c7fea2\" (UID: \"060d361e-76fd-4e2e-9626-7b6796c7fea2\") "
Sep 16 19:30:01.374893 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.374822    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djnb7\" (UniqueName: \"kubernetes.io/projected/060d361e-76fd-4e2e-9626-7b6796c7fea2-kube-api-access-djnb7\") pod \"060d361e-76fd-4e2e-9626-7b6796c7fea2\" (UID: \"060d361e-76fd-4e2e-9626-7b6796c7fea2\") "
Sep 16 19:30:01.377135 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.377108    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060d361e-76fd-4e2e-9626-7b6796c7fea2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "060d361e-76fd-4e2e-9626-7b6796c7fea2" (UID: "060d361e-76fd-4e2e-9626-7b6796c7fea2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:01.377249 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.377176    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060d361e-76fd-4e2e-9626-7b6796c7fea2-kube-api-access-djnb7" (OuterVolumeSpecName: "kube-api-access-djnb7") pod "060d361e-76fd-4e2e-9626-7b6796c7fea2" (UID: "060d361e-76fd-4e2e-9626-7b6796c7fea2"). InnerVolumeSpecName "kube-api-access-djnb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:01.475629 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.475586    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d361e-76fd-4e2e-9626-7b6796c7fea2-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:30:01.475629 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.475621    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djnb7\" (UniqueName: \"kubernetes.io/projected/060d361e-76fd-4e2e-9626-7b6796c7fea2-kube-api-access-djnb7\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:30:01.592150 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.592111    2590 generic.go:356] "Generic (PLEG): container finished" podID="060d361e-76fd-4e2e-9626-7b6796c7fea2" containerID="36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df" exitCode=0
Sep 16 19:30:01.592317 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.592154    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"
Sep 16 19:30:01.592317 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.592155    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" event={"ID":"060d361e-76fd-4e2e-9626-7b6796c7fea2","Type":"ContainerDied","Data":"36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df"}
Sep 16 19:30:01.592317 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.592284    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8" event={"ID":"060d361e-76fd-4e2e-9626-7b6796c7fea2","Type":"ContainerDied","Data":"d37d6383aab9bda47453ce009e46cb59734b28b801779b6ce8ef8eafe12254c9"}
Sep 16 19:30:01.592317 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.592313    2590 scope.go:117] "RemoveContainer" containerID="36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df"
Sep 16 19:30:01.601471 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.601448    2590 scope.go:117] "RemoveContainer" containerID="36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df"
Sep 16 19:30:01.601772 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:30:01.601744    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df\": container with ID starting with 36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df not found: ID does not exist" containerID="36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df"
Sep 16 19:30:01.601832 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.601781    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df"} err="failed to get container status \"36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df\": rpc error: code = NotFound desc = could not find container \"36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df\": container with ID starting with 36784146a6e2d86fcc1162938217ca67ab9b24e3212dd7a3bb02a71c814e91df not found: ID does not exist"
Sep 16 19:30:01.616290 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.616252    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"]
Sep 16 19:30:01.620547 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:01.620517    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schebcnd8"]
Sep 16 19:30:03.025199 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:03.025163    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060d361e-76fd-4e2e-9626-7b6796c7fea2" path="/var/lib/kubelet/pods/060d361e-76fd-4e2e-9626-7b6796c7fea2/volumes"
Sep 16 19:30:04.072690 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:04.072658    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:04.196249 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:04.196218    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:10.442640 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:10.442605    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:13.468635 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.468602    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"]
Sep 16 19:30:13.469013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.468864    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060d361e-76fd-4e2e-9626-7b6796c7fea2" containerName="main"
Sep 16 19:30:13.469013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.468874    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="060d361e-76fd-4e2e-9626-7b6796c7fea2" containerName="main"
Sep 16 19:30:13.469013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.468902    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main"
Sep 16 19:30:13.469013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.468908    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main"
Sep 16 19:30:13.469013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.468919    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="storage-initializer"
Sep 16 19:30:13.469013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.468925    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="storage-initializer"
Sep 16 19:30:13.469013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.468980    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="905e0db1-e120-45f5-8ddc-54685e7a50dc" containerName="main"
Sep 16 19:30:13.469013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.468987    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="060d361e-76fd-4e2e-9626-7b6796c7fea2" containerName="main"
Sep 16 19:30:13.473142 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.473119    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.476150 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.476126    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-944gn\""
Sep 16 19:30:13.476267 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.476126    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 19:30:13.476267 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.476140    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 19:30:13.476267 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.476157    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:30:13.476570 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.476553    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 19:30:13.476613 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.476574    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 19:30:13.493301 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.493273    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"]
Sep 16 19:30:13.504975 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.504945    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:13.571429 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.571396    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.571429 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.571433    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.571647 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.571503    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.571647 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.571544    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.571647 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.571601    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860d2b-0195-4a0a-87ab-064d6166479b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.571647 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.571631    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.571778 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.571652    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ggcw\" (UniqueName: \"kubernetes.io/projected/51860d2b-0195-4a0a-87ab-064d6166479b-kube-api-access-8ggcw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.571778 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.571671    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51860d2b-0195-4a0a-87ab-064d6166479b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672231 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672172    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672231 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672225    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ggcw\" (UniqueName: \"kubernetes.io/projected/51860d2b-0195-4a0a-87ab-064d6166479b-kube-api-access-8ggcw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672231 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672244    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51860d2b-0195-4a0a-87ab-064d6166479b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672522 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672262    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672522 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672280    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672522 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672325    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672522 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672357    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672522 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672406    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860d2b-0195-4a0a-87ab-064d6166479b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672809 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672771    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672873 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672802    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.672873 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672829    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.673031 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.672915    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.673220 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.673200    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51860d2b-0195-4a0a-87ab-064d6166479b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.675088 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.675063    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.675208 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.675189    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860d2b-0195-4a0a-87ab-064d6166479b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.683144 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.683122    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ggcw\" (UniqueName: \"kubernetes.io/projected/51860d2b-0195-4a0a-87ab-064d6166479b-kube-api-access-8ggcw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.714724 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.714686    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"]
Sep 16 19:30:13.719039 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.718965    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:13.721587 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.721565    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-sctjn\""
Sep 16 19:30:13.727630 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.727605    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"]
Sep 16 19:30:13.741388 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.741357    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:13.783270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.783236    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:13.873708 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.873674    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49b9g\" (UniqueName: \"kubernetes.io/projected/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-kube-api-access-49b9g\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz\" (UID: \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:13.873708 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.873722    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz\" (UID: \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:13.975158 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.975047    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49b9g\" (UniqueName: \"kubernetes.io/projected/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-kube-api-access-49b9g\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz\" (UID: \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:13.975158 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.975104    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz\" (UID: \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:13.977795 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.977774    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz\" (UID: \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:13.984452 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:13.984430    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49b9g\" (UniqueName: \"kubernetes.io/projected/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-kube-api-access-49b9g\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz\" (UID: \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:14.030690 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:14.030655    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:14.063356 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:14.063327    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"]
Sep 16 19:30:14.065389 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:30:14.065355    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51860d2b_0195_4a0a_87ab_064d6166479b.slice/crio-61844d7d9e39424238a70808b938b6047f60cd2ce7ec81494606328501cc3602 WatchSource:0}: Error finding container 61844d7d9e39424238a70808b938b6047f60cd2ce7ec81494606328501cc3602: Status 404 returned error can't find the container with id 61844d7d9e39424238a70808b938b6047f60cd2ce7ec81494606328501cc3602
Sep 16 19:30:14.316813 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:14.316781    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"]
Sep 16 19:30:14.318389 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:30:14.318345    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79f5ecb_051e_4f46_98e1_ebb9376d8e0e.slice/crio-cc5c96092f8314551a896ea5b8f6f2207e0d45de12fa8dbe40d79c290a9b86ce WatchSource:0}: Error finding container cc5c96092f8314551a896ea5b8f6f2207e0d45de12fa8dbe40d79c290a9b86ce: Status 404 returned error can't find the container with id cc5c96092f8314551a896ea5b8f6f2207e0d45de12fa8dbe40d79c290a9b86ce
Sep 16 19:30:14.652665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:14.652555    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" event={"ID":"51860d2b-0195-4a0a-87ab-064d6166479b","Type":"ContainerStarted","Data":"b07408e97899cfd4c78ffef0e1bfc3f93c0363f4197c83e9fd7f43b347369337"}
Sep 16 19:30:14.652665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:14.652634    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" event={"ID":"51860d2b-0195-4a0a-87ab-064d6166479b","Type":"ContainerStarted","Data":"61844d7d9e39424238a70808b938b6047f60cd2ce7ec81494606328501cc3602"}
Sep 16 19:30:14.654303 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:14.654276    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" event={"ID":"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e","Type":"ContainerStarted","Data":"597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d"}
Sep 16 19:30:14.654430 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:14.654321    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" event={"ID":"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e","Type":"ContainerStarted","Data":"cc5c96092f8314551a896ea5b8f6f2207e0d45de12fa8dbe40d79c290a9b86ce"}
Sep 16 19:30:14.654430 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:14.654343    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:14.701942 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:14.701858    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" podStartSLOduration=1.701831888 podStartE2EDuration="1.701831888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:14.699811259 +0000 UTC m=+3030.224270093" watchObservedRunningTime="2026-09-16 19:30:14.701831888 +0000 UTC m=+3030.226290724"
Sep 16 19:30:16.779445 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:16.779409    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:16.874197 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:16.874162    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:17.670945 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:17.670913    2590 generic.go:356] "Generic (PLEG): container finished" podID="51860d2b-0195-4a0a-87ab-064d6166479b" containerID="b07408e97899cfd4c78ffef0e1bfc3f93c0363f4197c83e9fd7f43b347369337" exitCode=0
Sep 16 19:30:17.671129 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:17.670998    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" event={"ID":"51860d2b-0195-4a0a-87ab-064d6166479b","Type":"ContainerDied","Data":"b07408e97899cfd4c78ffef0e1bfc3f93c0363f4197c83e9fd7f43b347369337"}
Sep 16 19:30:18.678921 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:18.678869    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" event={"ID":"51860d2b-0195-4a0a-87ab-064d6166479b","Type":"ContainerStarted","Data":"c8e70436a7dd5327351e6b4defc7d5e7f356cc17d3cfd9c4cdee224931e44637"}
Sep 16 19:30:18.711954 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:18.711899    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podStartSLOduration=2.910712479 podStartE2EDuration="5.711859536s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.801147057s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:18.709701832 +0000 UTC m=+3034.234160668" watchObservedRunningTime="2026-09-16 19:30:18.711859536 +0000 UTC m=+3034.236318369"
Sep 16 19:30:18.784001 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:18.783970    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:19.308586 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:19.308553    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:23.783819 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:23.783778    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:30:23.785474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:23.785429    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:30:33.784186 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:33.784132    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:30:43.784040 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:43.783992    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:30:45.662183 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:45.662095    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:30:53.784130 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:30:53.784080    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:03.784490 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:03.784442    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:13.783708 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:13.783657    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:18.179479 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:18.179441    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:18.244372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:18.244337    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:23.783795 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:23.783755    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:33.794078 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:33.794042    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:31:33.802402 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:33.802370    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:31:35.473104 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:35.473075    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"]
Sep 16 19:31:35.480429 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:35.480369    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"]
Sep 16 19:31:35.481021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:35.480996    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:35.574578 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:35.574546    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:35.661857 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:35.661821    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:45.661430 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:45.661391    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:50.499050 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:50.499007    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main" containerID="cri-o://c8e70436a7dd5327351e6b4defc7d5e7f356cc17d3cfd9c4cdee224931e44637" gracePeriod=45
Sep 16 19:31:50.500817 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:50.500777    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerName="main" containerID="cri-o://597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d" gracePeriod=45
Sep 16 19:31:50.795086 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:50.795046    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:51.101008 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.100969    2590 generic.go:356] "Generic (PLEG): container finished" podID="51860d2b-0195-4a0a-87ab-064d6166479b" containerID="c8e70436a7dd5327351e6b4defc7d5e7f356cc17d3cfd9c4cdee224931e44637" exitCode=0
Sep 16 19:31:51.101168 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.101008    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" event={"ID":"51860d2b-0195-4a0a-87ab-064d6166479b","Type":"ContainerDied","Data":"c8e70436a7dd5327351e6b4defc7d5e7f356cc17d3cfd9c4cdee224931e44637"}
Sep 16 19:31:51.152719 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.152687    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:31:51.282203 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282163    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-home\") pod \"51860d2b-0195-4a0a-87ab-064d6166479b\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") "
Sep 16 19:31:51.282203 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282212    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860d2b-0195-4a0a-87ab-064d6166479b-tls-certs\") pod \"51860d2b-0195-4a0a-87ab-064d6166479b\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") "
Sep 16 19:31:51.282474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282236    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ggcw\" (UniqueName: \"kubernetes.io/projected/51860d2b-0195-4a0a-87ab-064d6166479b-kube-api-access-8ggcw\") pod \"51860d2b-0195-4a0a-87ab-064d6166479b\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") "
Sep 16 19:31:51.282474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282260    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-model-cache\") pod \"51860d2b-0195-4a0a-87ab-064d6166479b\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") "
Sep 16 19:31:51.282474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282282    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-tmp-dir\") pod \"51860d2b-0195-4a0a-87ab-064d6166479b\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") "
Sep 16 19:31:51.282474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282315    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-dshm\") pod \"51860d2b-0195-4a0a-87ab-064d6166479b\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") "
Sep 16 19:31:51.282474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282342    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-kserve-provision-location\") pod \"51860d2b-0195-4a0a-87ab-064d6166479b\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") "
Sep 16 19:31:51.282474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282372    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51860d2b-0195-4a0a-87ab-064d6166479b-cabundle-cert\") pod \"51860d2b-0195-4a0a-87ab-064d6166479b\" (UID: \"51860d2b-0195-4a0a-87ab-064d6166479b\") "
Sep 16 19:31:51.282785 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282630    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-model-cache" (OuterVolumeSpecName: "model-cache") pod "51860d2b-0195-4a0a-87ab-064d6166479b" (UID: "51860d2b-0195-4a0a-87ab-064d6166479b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:51.282785 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.282747    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-home" (OuterVolumeSpecName: "home") pod "51860d2b-0195-4a0a-87ab-064d6166479b" (UID: "51860d2b-0195-4a0a-87ab-064d6166479b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:51.283092 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.283061    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51860d2b-0195-4a0a-87ab-064d6166479b" (UID: "51860d2b-0195-4a0a-87ab-064d6166479b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:51.283092 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.283078    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51860d2b-0195-4a0a-87ab-064d6166479b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51860d2b-0195-4a0a-87ab-064d6166479b" (UID: "51860d2b-0195-4a0a-87ab-064d6166479b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:51.285106 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.285075    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-dshm" (OuterVolumeSpecName: "dshm") pod "51860d2b-0195-4a0a-87ab-064d6166479b" (UID: "51860d2b-0195-4a0a-87ab-064d6166479b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:51.285215 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.285134    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51860d2b-0195-4a0a-87ab-064d6166479b-kube-api-access-8ggcw" (OuterVolumeSpecName: "kube-api-access-8ggcw") pod "51860d2b-0195-4a0a-87ab-064d6166479b" (UID: "51860d2b-0195-4a0a-87ab-064d6166479b"). InnerVolumeSpecName "kube-api-access-8ggcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:51.285215 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.285159    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51860d2b-0195-4a0a-87ab-064d6166479b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51860d2b-0195-4a0a-87ab-064d6166479b" (UID: "51860d2b-0195-4a0a-87ab-064d6166479b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:51.307229 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.307191    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51860d2b-0195-4a0a-87ab-064d6166479b" (UID: "51860d2b-0195-4a0a-87ab-064d6166479b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:51.383839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.383744    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:51.383839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.383781    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:51.383839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.383793    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51860d2b-0195-4a0a-87ab-064d6166479b-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:51.383839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.383802    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:51.383839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.383810    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860d2b-0195-4a0a-87ab-064d6166479b-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:51.383839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.383818    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ggcw\" (UniqueName: \"kubernetes.io/projected/51860d2b-0195-4a0a-87ab-064d6166479b-kube-api-access-8ggcw\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:51.383839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.383826    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:51.383839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:51.383834    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860d2b-0195-4a0a-87ab-064d6166479b-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:52.106481 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:52.106444    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"
Sep 16 19:31:52.106481 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:52.106460    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg" event={"ID":"51860d2b-0195-4a0a-87ab-064d6166479b","Type":"ContainerDied","Data":"61844d7d9e39424238a70808b938b6047f60cd2ce7ec81494606328501cc3602"}
Sep 16 19:31:52.107103 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:52.106524    2590 scope.go:117] "RemoveContainer" containerID="c8e70436a7dd5327351e6b4defc7d5e7f356cc17d3cfd9c4cdee224931e44637"
Sep 16 19:31:52.115495 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:52.115475    2590 scope.go:117] "RemoveContainer" containerID="b07408e97899cfd4c78ffef0e1bfc3f93c0363f4197c83e9fd7f43b347369337"
Sep 16 19:31:52.128955 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:52.128932    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"]
Sep 16 19:31:52.132991 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:52.132970    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85cf9658d-5g6rg"]
Sep 16 19:31:53.025008 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:53.024975    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" path="/var/lib/kubelet/pods/51860d2b-0195-4a0a-87ab-064d6166479b/volumes"
Sep 16 19:31:55.661619 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:55.661582    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:55.662047 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:31:55.661699    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:32:00.516534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:00.516498    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:05.661459 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:05.661426    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:08.283483 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:08.283443    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:08.384239 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:08.384201    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:08.487423 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:08.487387    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:08.570610 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:08.570519    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:12.372910 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:12.372863    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:14.742517 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:14.742482    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:15.661502 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:15.661469    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:16.351363 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:16.351328    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:17.382670 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:17.382631    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:20.597981 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.597943    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:20.755175 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.755150    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:32:20.824173 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.824142    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49b9g\" (UniqueName: \"kubernetes.io/projected/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-kube-api-access-49b9g\") pod \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\" (UID: \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\") "
Sep 16 19:32:20.824173 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.824178    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-tls-certs\") pod \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\" (UID: \"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e\") "
Sep 16 19:32:20.826452 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.826416    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-kube-api-access-49b9g" (OuterVolumeSpecName: "kube-api-access-49b9g") pod "c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" (UID: "c79f5ecb-051e-4f46-98e1-ebb9376d8e0e"). InnerVolumeSpecName "kube-api-access-49b9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:20.826594 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.826578    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" (UID: "c79f5ecb-051e-4f46-98e1-ebb9376d8e0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:20.858961 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.858920    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"]
Sep 16 19:32:20.859272 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.859254    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main"
Sep 16 19:32:20.859363 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.859275    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main"
Sep 16 19:32:20.859363 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.859286    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="storage-initializer"
Sep 16 19:32:20.859363 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.859294    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="storage-initializer"
Sep 16 19:32:20.859363 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.859313    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerName="main"
Sep 16 19:32:20.859363 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.859322    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerName="main"
Sep 16 19:32:20.859606 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.859396    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerName="main"
Sep 16 19:32:20.859606 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.859415    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="51860d2b-0195-4a0a-87ab-064d6166479b" containerName="main"
Sep 16 19:32:20.862309 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.862286    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:20.865839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.865811    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-znzxm\""
Sep 16 19:32:20.865839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.865829    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:32:20.866079 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.865837    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:32:20.866079 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.865811    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:32:20.866079 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.865872    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-d5kwm\""
Sep 16 19:32:20.874127 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.874105    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"]
Sep 16 19:32:20.925615 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.925584    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwqj8\" (UniqueName: \"kubernetes.io/projected/7801aab7-8e0e-46f3-8165-35a96d949f38-kube-api-access-qwqj8\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld\" (UID: \"7801aab7-8e0e-46f3-8165-35a96d949f38\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:20.925788 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.925644    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7801aab7-8e0e-46f3-8165-35a96d949f38-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld\" (UID: \"7801aab7-8e0e-46f3-8165-35a96d949f38\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:20.925788 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.925695    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49b9g\" (UniqueName: \"kubernetes.io/projected/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-kube-api-access-49b9g\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:32:20.925788 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.925710    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:32:20.939735 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:20.939705    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:21.026384 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.026358    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwqj8\" (UniqueName: \"kubernetes.io/projected/7801aab7-8e0e-46f3-8165-35a96d949f38-kube-api-access-qwqj8\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld\" (UID: \"7801aab7-8e0e-46f3-8165-35a96d949f38\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:21.026557 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.026411    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7801aab7-8e0e-46f3-8165-35a96d949f38-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld\" (UID: \"7801aab7-8e0e-46f3-8165-35a96d949f38\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:21.029002 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.028982    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7801aab7-8e0e-46f3-8165-35a96d949f38-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld\" (UID: \"7801aab7-8e0e-46f3-8165-35a96d949f38\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:21.038601 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.038575    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwqj8\" (UniqueName: \"kubernetes.io/projected/7801aab7-8e0e-46f3-8165-35a96d949f38-kube-api-access-qwqj8\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld\" (UID: \"7801aab7-8e0e-46f3-8165-35a96d949f38\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:21.040937 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.040913    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:21.143548 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.143448    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:21.173014 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.172982    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:21.236861 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.236828    2590 generic.go:356] "Generic (PLEG): container finished" podID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" containerID="597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d" exitCode=0
Sep 16 19:32:21.237054 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.236871    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"
Sep 16 19:32:21.237054 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.236922    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" event={"ID":"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e","Type":"ContainerDied","Data":"597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d"}
Sep 16 19:32:21.237054 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.236997    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz" event={"ID":"c79f5ecb-051e-4f46-98e1-ebb9376d8e0e","Type":"ContainerDied","Data":"cc5c96092f8314551a896ea5b8f6f2207e0d45de12fa8dbe40d79c290a9b86ce"}
Sep 16 19:32:21.237054 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.237034    2590 scope.go:117] "RemoveContainer" containerID="597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d"
Sep 16 19:32:21.246730 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.246710    2590 scope.go:117] "RemoveContainer" containerID="597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d"
Sep 16 19:32:21.247074 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:32:21.247050    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d\": container with ID starting with 597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d not found: ID does not exist" containerID="597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d"
Sep 16 19:32:21.247132 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.247085    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d"} err="failed to get container status \"597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d\": rpc error: code = NotFound desc = could not find container \"597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d\": container with ID starting with 597f41fc27b993a37457836473a9dd659b491678100e7dcfdc2c2d671af22e2d not found: ID does not exist"
Sep 16 19:32:21.254684 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.254663    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"]
Sep 16 19:32:21.262674 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.262654    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesctpz"]
Sep 16 19:32:21.444110 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:21.444085    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"]
Sep 16 19:32:21.446088 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:32:21.446063    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7801aab7_8e0e_46f3_8165_35a96d949f38.slice/crio-725d88aa0f735542685d6aae23d3448ea58a311871240c8bb4123b439f5d037c WatchSource:0}: Error finding container 725d88aa0f735542685d6aae23d3448ea58a311871240c8bb4123b439f5d037c: Status 404 returned error can't find the container with id 725d88aa0f735542685d6aae23d3448ea58a311871240c8bb4123b439f5d037c
Sep 16 19:32:22.248797 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:22.248758    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" event={"ID":"7801aab7-8e0e-46f3-8165-35a96d949f38","Type":"ContainerStarted","Data":"1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd"}
Sep 16 19:32:22.249310 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:22.248862    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:22.249310 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:22.248905    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" event={"ID":"7801aab7-8e0e-46f3-8165-35a96d949f38","Type":"ContainerStarted","Data":"725d88aa0f735542685d6aae23d3448ea58a311871240c8bb4123b439f5d037c"}
Sep 16 19:32:22.269498 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:22.269445    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" podStartSLOduration=2.269429804 podStartE2EDuration="2.269429804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:22.268169968 +0000 UTC m=+3157.792628814" watchObservedRunningTime="2026-09-16 19:32:22.269429804 +0000 UTC m=+3157.793888651"
Sep 16 19:32:23.025846 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:23.025808    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c79f5ecb-051e-4f46-98e1-ebb9376d8e0e" path="/var/lib/kubelet/pods/c79f5ecb-051e-4f46-98e1-ebb9376d8e0e/volumes"
Sep 16 19:32:25.525063 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:25.525031    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:25.625574 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:25.625538    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:25.728540 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:25.728505    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:27.992793 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:27.992762    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:28.106243 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:28.106200    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:32.051102 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:32.051060    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:34.213924 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:34.213861    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:34.313244 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:34.313208    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:34.773805 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:34.773775    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:34.884687 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:34.884651    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:34.988064 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:34.988021    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:35.388019 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:35.387988    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:37.924951 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:37.924911    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:38.025567 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:38.025538    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:39.107092 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:39.107044    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:39.201824 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:39.201786    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:41.043130 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:41.043097    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:41.143146 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:41.143114    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:44.172435 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:44.172399    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:44.430625 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:44.430534    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:44.520305 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:44.520271    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:45.980868 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:45.980834    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:46.084548 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:46.084510    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:49.465965 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:49.465934    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:49.563822 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:49.563785    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:51.239065 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:51.239025    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:51.301890 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:51.301853    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:53.256232 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:53.256200    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:32:55.708539 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:55.708503    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:55.762425 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:55.762397    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:57.261247 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.261213    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"]
Sep 16 19:32:57.264387 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.264364    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.266848 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.266820    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:32:57.266990 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.266864    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:32:57.267077 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.267059    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:32:57.267136 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.267096    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-pg5rx\""
Sep 16 19:32:57.267221 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.267198    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:32:57.267284 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.267238    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:57.277795 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.277762    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"]
Sep 16 19:32:57.291752 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.291727    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:57.329130 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.329093    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-home\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.329341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.329150    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-kserve-provision-location\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.329341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.329202    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-dshm\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.329341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.329230    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-tmp-dir\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.329341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.329261    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805e6363-1525-449a-bb97-abd15cfbc667-tls-certs\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.329341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.329288    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gxjj\" (UniqueName: \"kubernetes.io/projected/805e6363-1525-449a-bb97-abd15cfbc667-kube-api-access-4gxjj\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.329341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.329330    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-model-cache\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.329585 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.329379    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805e6363-1525-449a-bb97-abd15cfbc667-cabundle-cert\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430261 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430224    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-kserve-provision-location\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430261 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430269    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-dshm\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430289    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-tmp-dir\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430305    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805e6363-1525-449a-bb97-abd15cfbc667-tls-certs\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430331    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gxjj\" (UniqueName: \"kubernetes.io/projected/805e6363-1525-449a-bb97-abd15cfbc667-kube-api-access-4gxjj\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430356    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-model-cache\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430411    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805e6363-1525-449a-bb97-abd15cfbc667-cabundle-cert\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430444    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-home\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430900 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430647    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-kserve-provision-location\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.430900 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.430749    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-tmp-dir\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.431045 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.431022    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-model-cache\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.431089 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.431058    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-home\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.431178 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.431161    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805e6363-1525-449a-bb97-abd15cfbc667-cabundle-cert\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.433376 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.433355    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-dshm\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.433749 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.433728    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805e6363-1525-449a-bb97-abd15cfbc667-tls-certs\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.438026 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.438007    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gxjj\" (UniqueName: \"kubernetes.io/projected/805e6363-1525-449a-bb97-abd15cfbc667-kube-api-access-4gxjj\") pod \"stop-feature-test-kserve-58d658465-vc4lz\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.492619 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.488712    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:57.575057 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.575021    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:32:57.858532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:57.858414    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"]
Sep 16 19:32:57.860936 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:32:57.860850    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805e6363_1525_449a_bb97_abd15cfbc667.slice/crio-529e2be73c8e631c32750c9b2f155798361644d38fc8a26a49f15885229f8bb2 WatchSource:0}: Error finding container 529e2be73c8e631c32750c9b2f155798361644d38fc8a26a49f15885229f8bb2: Status 404 returned error can't find the container with id 529e2be73c8e631c32750c9b2f155798361644d38fc8a26a49f15885229f8bb2
Sep 16 19:32:58.423568 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:58.423530    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" event={"ID":"805e6363-1525-449a-bb97-abd15cfbc667","Type":"ContainerStarted","Data":"261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb"}
Sep 16 19:32:58.424222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:58.424199    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" event={"ID":"805e6363-1525-449a-bb97-abd15cfbc667","Type":"ContainerStarted","Data":"529e2be73c8e631c32750c9b2f155798361644d38fc8a26a49f15885229f8bb2"}
Sep 16 19:32:58.828046 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:58.828008    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:58.931100 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:32:58.931067    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:01.444273 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:01.444238    2590 generic.go:356] "Generic (PLEG): container finished" podID="805e6363-1525-449a-bb97-abd15cfbc667" containerID="261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb" exitCode=0
Sep 16 19:33:01.444787 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:01.444318    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" event={"ID":"805e6363-1525-449a-bb97-abd15cfbc667","Type":"ContainerDied","Data":"261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb"}
Sep 16 19:33:02.452673 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:02.452632    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" event={"ID":"805e6363-1525-449a-bb97-abd15cfbc667","Type":"ContainerStarted","Data":"7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8"}
Sep 16 19:33:02.479576 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:02.479516    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podStartSLOduration=2.81556158 podStartE2EDuration="5.479499698s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.663938118s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:02.477153735 +0000 UTC m=+3198.001612570" watchObservedRunningTime="2026-09-16 19:33:02.479499698 +0000 UTC m=+3198.003958531"
Sep 16 19:33:02.575711 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:02.575668    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:33:04.669633 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:04.669588    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:07.576140 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:07.576106    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:33:07.577853 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:07.577824    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:33:17.576256 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:17.576207    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:33:27.575466 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:27.575421    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:33:37.576475 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:37.576380    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:33:47.575606 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:47.575550    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:33:57.576074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:33:57.576023    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:34:07.575855 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:07.575807    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:34:17.575820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:17.575770    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:34:26.024497 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:26.024456    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:26.026071 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:26.026041    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"]
Sep 16 19:34:26.149700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:26.149665    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:27.576093 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:27.576046    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:34:33.256479 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:33.256433    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" podUID="7801aab7-8e0e-46f3-8165-35a96d949f38" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:37.576166 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:37.576121    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:34:41.046815 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:41.046742    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" podUID="7801aab7-8e0e-46f3-8165-35a96d949f38" containerName="main" containerID="cri-o://1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd" gracePeriod=45
Sep 16 19:34:43.255732 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:43.255691    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" podUID="7801aab7-8e0e-46f3-8165-35a96d949f38" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:47.576075 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:47.576032    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:34:53.256622 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:53.256580    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" podUID="7801aab7-8e0e-46f3-8165-35a96d949f38" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:53.257177 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:53.256704    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:34:57.585950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:57.585910    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:34:57.593545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:57.593520    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:34:59.540369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:34:59.540334    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"]
Sep 16 19:35:00.297965 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:00.297933    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:00.397945 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:00.397865    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:03.256374 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:03.256330    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" podUID="7801aab7-8e0e-46f3-8165-35a96d949f38" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:11.290984 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:11.290957    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:35:11.405870 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:11.405776    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7801aab7-8e0e-46f3-8165-35a96d949f38-tls-certs\") pod \"7801aab7-8e0e-46f3-8165-35a96d949f38\" (UID: \"7801aab7-8e0e-46f3-8165-35a96d949f38\") "
Sep 16 19:35:11.406044 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:11.405911    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwqj8\" (UniqueName: \"kubernetes.io/projected/7801aab7-8e0e-46f3-8165-35a96d949f38-kube-api-access-qwqj8\") pod \"7801aab7-8e0e-46f3-8165-35a96d949f38\" (UID: \"7801aab7-8e0e-46f3-8165-35a96d949f38\") "
Sep 16 19:35:11.408289 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:11.408255    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7801aab7-8e0e-46f3-8165-35a96d949f38-kube-api-access-qwqj8" (OuterVolumeSpecName: "kube-api-access-qwqj8") pod "7801aab7-8e0e-46f3-8165-35a96d949f38" (UID: "7801aab7-8e0e-46f3-8165-35a96d949f38"). InnerVolumeSpecName "kube-api-access-qwqj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:11.408289 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:11.408257    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7801aab7-8e0e-46f3-8165-35a96d949f38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7801aab7-8e0e-46f3-8165-35a96d949f38" (UID: "7801aab7-8e0e-46f3-8165-35a96d949f38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:11.506982 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:11.506942    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7801aab7-8e0e-46f3-8165-35a96d949f38-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:11.506982 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:11.506974    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwqj8\" (UniqueName: \"kubernetes.io/projected/7801aab7-8e0e-46f3-8165-35a96d949f38-kube-api-access-qwqj8\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:12.046176 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:12.046144    2590 generic.go:356] "Generic (PLEG): container finished" podID="7801aab7-8e0e-46f3-8165-35a96d949f38" containerID="1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd" exitCode=0
Sep 16 19:35:12.046355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:12.046190    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"
Sep 16 19:35:12.046355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:12.046232    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" event={"ID":"7801aab7-8e0e-46f3-8165-35a96d949f38","Type":"ContainerDied","Data":"1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd"}
Sep 16 19:35:12.046355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:12.046287    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld" event={"ID":"7801aab7-8e0e-46f3-8165-35a96d949f38","Type":"ContainerDied","Data":"725d88aa0f735542685d6aae23d3448ea58a311871240c8bb4123b439f5d037c"}
Sep 16 19:35:12.046355 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:12.046317    2590 scope.go:117] "RemoveContainer" containerID="1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd"
Sep 16 19:35:12.055713 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:12.055695    2590 scope.go:117] "RemoveContainer" containerID="1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd"
Sep 16 19:35:12.056029 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:35:12.056008    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd\": container with ID starting with 1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd not found: ID does not exist" containerID="1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd"
Sep 16 19:35:12.056093 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:12.056040    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd"} err="failed to get container status \"1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd\": rpc error: code = NotFound desc = could not find container \"1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd\": container with ID starting with 1a4eaf61098bbc722d0d120b70790890c5edf0874d9669dbcf159dab1aba9edd not found: ID does not exist"
Sep 16 19:35:12.068997 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:12.068970    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"]
Sep 16 19:35:12.074062 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:12.074042    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rhhld"]
Sep 16 19:35:13.025623 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:13.025593    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7801aab7-8e0e-46f3-8165-35a96d949f38" path="/var/lib/kubelet/pods/7801aab7-8e0e-46f3-8165-35a96d949f38/volumes"
Sep 16 19:35:14.563482 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:14.563436    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main" containerID="cri-o://7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8" gracePeriod=45
Sep 16 19:35:15.314225 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.314198    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:35:15.339497 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.339466    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gxjj\" (UniqueName: \"kubernetes.io/projected/805e6363-1525-449a-bb97-abd15cfbc667-kube-api-access-4gxjj\") pod \"805e6363-1525-449a-bb97-abd15cfbc667\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") "
Sep 16 19:35:15.339671 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.339520    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-model-cache\") pod \"805e6363-1525-449a-bb97-abd15cfbc667\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") "
Sep 16 19:35:15.339671 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.339551    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805e6363-1525-449a-bb97-abd15cfbc667-tls-certs\") pod \"805e6363-1525-449a-bb97-abd15cfbc667\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") "
Sep 16 19:35:15.339671 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.339582    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-tmp-dir\") pod \"805e6363-1525-449a-bb97-abd15cfbc667\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") "
Sep 16 19:35:15.339671 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.339610    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-home\") pod \"805e6363-1525-449a-bb97-abd15cfbc667\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") "
Sep 16 19:35:15.339671 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.339668    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-kserve-provision-location\") pod \"805e6363-1525-449a-bb97-abd15cfbc667\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") "
Sep 16 19:35:15.340038 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.339706    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-dshm\") pod \"805e6363-1525-449a-bb97-abd15cfbc667\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") "
Sep 16 19:35:15.340038 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.339735    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805e6363-1525-449a-bb97-abd15cfbc667-cabundle-cert\") pod \"805e6363-1525-449a-bb97-abd15cfbc667\" (UID: \"805e6363-1525-449a-bb97-abd15cfbc667\") "
Sep 16 19:35:15.340384 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.340313    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805e6363-1525-449a-bb97-abd15cfbc667-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "805e6363-1525-449a-bb97-abd15cfbc667" (UID: "805e6363-1525-449a-bb97-abd15cfbc667"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:15.340694 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.340651    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-model-cache" (OuterVolumeSpecName: "model-cache") pod "805e6363-1525-449a-bb97-abd15cfbc667" (UID: "805e6363-1525-449a-bb97-abd15cfbc667"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:15.341070 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.341042    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-home" (OuterVolumeSpecName: "home") pod "805e6363-1525-449a-bb97-abd15cfbc667" (UID: "805e6363-1525-449a-bb97-abd15cfbc667"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:15.342629 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.342597    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805e6363-1525-449a-bb97-abd15cfbc667-kube-api-access-4gxjj" (OuterVolumeSpecName: "kube-api-access-4gxjj") pod "805e6363-1525-449a-bb97-abd15cfbc667" (UID: "805e6363-1525-449a-bb97-abd15cfbc667"). InnerVolumeSpecName "kube-api-access-4gxjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:15.343251 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.343228    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-dshm" (OuterVolumeSpecName: "dshm") pod "805e6363-1525-449a-bb97-abd15cfbc667" (UID: "805e6363-1525-449a-bb97-abd15cfbc667"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:15.343505 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.343484    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805e6363-1525-449a-bb97-abd15cfbc667-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "805e6363-1525-449a-bb97-abd15cfbc667" (UID: "805e6363-1525-449a-bb97-abd15cfbc667"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:15.354273 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.354203    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "805e6363-1525-449a-bb97-abd15cfbc667" (UID: "805e6363-1525-449a-bb97-abd15cfbc667"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:15.367530 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.367500    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "805e6363-1525-449a-bb97-abd15cfbc667" (UID: "805e6363-1525-449a-bb97-abd15cfbc667"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:15.440992 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.440954    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gxjj\" (UniqueName: \"kubernetes.io/projected/805e6363-1525-449a-bb97-abd15cfbc667-kube-api-access-4gxjj\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:15.440992 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.440989    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:15.440992 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.441000    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805e6363-1525-449a-bb97-abd15cfbc667-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:15.441207 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.441030    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:15.441207 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.441038    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:15.441207 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.441047    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:15.441207 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.441055    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805e6363-1525-449a-bb97-abd15cfbc667-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:15.441207 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:15.441063    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805e6363-1525-449a-bb97-abd15cfbc667-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:16.067127 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.067087    2590 generic.go:356] "Generic (PLEG): container finished" podID="805e6363-1525-449a-bb97-abd15cfbc667" containerID="7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8" exitCode=0
Sep 16 19:35:16.067584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.067139    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"
Sep 16 19:35:16.067584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.067143    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" event={"ID":"805e6363-1525-449a-bb97-abd15cfbc667","Type":"ContainerDied","Data":"7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8"}
Sep 16 19:35:16.067584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.067203    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz" event={"ID":"805e6363-1525-449a-bb97-abd15cfbc667","Type":"ContainerDied","Data":"529e2be73c8e631c32750c9b2f155798361644d38fc8a26a49f15885229f8bb2"}
Sep 16 19:35:16.067584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.067231    2590 scope.go:117] "RemoveContainer" containerID="7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8"
Sep 16 19:35:16.076413 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.076391    2590 scope.go:117] "RemoveContainer" containerID="261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb"
Sep 16 19:35:16.084348 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.084327    2590 scope.go:117] "RemoveContainer" containerID="7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8"
Sep 16 19:35:16.084635 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:35:16.084616    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8\": container with ID starting with 7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8 not found: ID does not exist" containerID="7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8"
Sep 16 19:35:16.084680 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.084644    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8"} err="failed to get container status \"7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8\": rpc error: code = NotFound desc = could not find container \"7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8\": container with ID starting with 7c219fa753d478e71f12e4ba6460b2754f86b2fcedee10c213735363f04262f8 not found: ID does not exist"
Sep 16 19:35:16.084680 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.084664    2590 scope.go:117] "RemoveContainer" containerID="261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb"
Sep 16 19:35:16.084956 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:35:16.084936    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb\": container with ID starting with 261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb not found: ID does not exist" containerID="261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb"
Sep 16 19:35:16.085022 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.084961    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb"} err="failed to get container status \"261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb\": rpc error: code = NotFound desc = could not find container \"261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb\": container with ID starting with 261d4b933e44c09b961c251011079b8836e84c5ae1b60dbb4962ea5342304bfb not found: ID does not exist"
Sep 16 19:35:16.090469 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.090445    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"]
Sep 16 19:35:16.099730 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.099705    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-vc4lz"]
Sep 16 19:35:16.257007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.256972    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:16.350489 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:16.350401    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:17.025202 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:17.025170    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="805e6363-1525-449a-bb97-abd15cfbc667" path="/var/lib/kubelet/pods/805e6363-1525-449a-bb97-abd15cfbc667/volumes"
Sep 16 19:35:21.323190 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:21.323153    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:30.962048 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.962015    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"]
Sep 16 19:35:30.962400 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.962355    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="storage-initializer"
Sep 16 19:35:30.962400 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.962370    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="storage-initializer"
Sep 16 19:35:30.962400 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.962379    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7801aab7-8e0e-46f3-8165-35a96d949f38" containerName="main"
Sep 16 19:35:30.962400 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.962385    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="7801aab7-8e0e-46f3-8165-35a96d949f38" containerName="main"
Sep 16 19:35:30.962400 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.962397    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main"
Sep 16 19:35:30.962400 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.962403    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main"
Sep 16 19:35:30.962603 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.962459    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="7801aab7-8e0e-46f3-8165-35a96d949f38" containerName="main"
Sep 16 19:35:30.962603 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.962469    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="805e6363-1525-449a-bb97-abd15cfbc667" containerName="main"
Sep 16 19:35:30.965402 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.965385    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:30.968051 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.968025    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:35:30.968205 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.968077    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:35:30.968205 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.968084    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:35:30.968205 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.968135    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-pg5rx\""
Sep 16 19:35:30.968205 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.968032    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:35:30.968394 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.968085    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:35:30.980642 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.980614    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"]
Sep 16 19:35:30.986915 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:30.986871    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:31.069014 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.068977    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-dshm\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.069014 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.069019    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-model-cache\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.069253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.069048    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-home\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.069253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.069066    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-cabundle-cert\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.069253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.069128    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tls-certs\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.069253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.069162    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvpvd\" (UniqueName: \"kubernetes.io/projected/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kube-api-access-mvpvd\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.069253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.069188    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tmp-dir\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.069253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.069220    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kserve-provision-location\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.130033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.130001    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"]
Sep 16 19:35:31.133231 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.133207    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:35:31.139919 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.139719    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-jh2n2\""
Sep 16 19:35:31.146805 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.146778    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"]
Sep 16 19:35:31.169847 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.169810    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-home\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.169847 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.169846    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-cabundle-cert\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.169867    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10681b13-4dc2-40b7-848d-1a059d3fee13-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b\" (UID: \"10681b13-4dc2-40b7-848d-1a059d3fee13\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:35:31.170084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.169930    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tls-certs\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.169955    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvpvd\" (UniqueName: \"kubernetes.io/projected/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kube-api-access-mvpvd\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.169977    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tmp-dir\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.170011    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kserve-provision-location\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.170050    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-dshm\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.170083    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-model-cache\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170424 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.170115    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqxb\" (UniqueName: \"kubernetes.io/projected/10681b13-4dc2-40b7-848d-1a059d3fee13-kube-api-access-rhqxb\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b\" (UID: \"10681b13-4dc2-40b7-848d-1a059d3fee13\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:35:31.170424 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.170315    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-home\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.170501    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tmp-dir\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170593 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.170572    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-model-cache\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170651 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.170590    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-cabundle-cert\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.170651 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.170625    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kserve-provision-location\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.172474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.172444    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-dshm\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.172607 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.172589    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tls-certs\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.177735 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.177712    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:31.178660 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.178626    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvpvd\" (UniqueName: \"kubernetes.io/projected/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kube-api-access-mvpvd\") pod \"stop-feature-test-kserve-58d658465-n8jv6\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.271496 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.271395    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqxb\" (UniqueName: \"kubernetes.io/projected/10681b13-4dc2-40b7-848d-1a059d3fee13-kube-api-access-rhqxb\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b\" (UID: \"10681b13-4dc2-40b7-848d-1a059d3fee13\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:35:31.271496 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.271445    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10681b13-4dc2-40b7-848d-1a059d3fee13-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b\" (UID: \"10681b13-4dc2-40b7-848d-1a059d3fee13\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:35:31.274164 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.274140    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10681b13-4dc2-40b7-848d-1a059d3fee13-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b\" (UID: \"10681b13-4dc2-40b7-848d-1a059d3fee13\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:35:31.275966 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.275947    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:31.280416 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.280321    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqxb\" (UniqueName: \"kubernetes.io/projected/10681b13-4dc2-40b7-848d-1a059d3fee13-kube-api-access-rhqxb\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b\" (UID: \"10681b13-4dc2-40b7-848d-1a059d3fee13\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:35:31.443817 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.443771    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:35:31.556270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.556245    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"]
Sep 16 19:35:31.557972 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:35:31.557942    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e9a0cf_bb50_4fd0_9495_03e803d0cddc.slice/crio-f4fd10b9014e9b366b28a2118c7fd1dbd0f1cdc0feb5f53c21c7e25698b391ef WatchSource:0}: Error finding container f4fd10b9014e9b366b28a2118c7fd1dbd0f1cdc0feb5f53c21c7e25698b391ef: Status 404 returned error can't find the container with id f4fd10b9014e9b366b28a2118c7fd1dbd0f1cdc0feb5f53c21c7e25698b391ef
Sep 16 19:35:31.717693 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:31.717661    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"]
Sep 16 19:35:31.719203 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:35:31.719169    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10681b13_4dc2_40b7_848d_1a059d3fee13.slice/crio-e79c46a4a045393bf6059e9a87621210d96e1208242c42eb01bc1bcef217878b WatchSource:0}: Error finding container e79c46a4a045393bf6059e9a87621210d96e1208242c42eb01bc1bcef217878b: Status 404 returned error can't find the container with id e79c46a4a045393bf6059e9a87621210d96e1208242c42eb01bc1bcef217878b
Sep 16 19:35:32.137782 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:32.137740    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" event={"ID":"53e9a0cf-bb50-4fd0-9495-03e803d0cddc","Type":"ContainerStarted","Data":"8101331f166b25871bfa305bb45ac0ced24f4fb08ef36279ea0aa203bdeb785d"}
Sep 16 19:35:32.138289 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:32.137816    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" event={"ID":"53e9a0cf-bb50-4fd0-9495-03e803d0cddc","Type":"ContainerStarted","Data":"f4fd10b9014e9b366b28a2118c7fd1dbd0f1cdc0feb5f53c21c7e25698b391ef"}
Sep 16 19:35:32.139543 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:32.139516    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" event={"ID":"10681b13-4dc2-40b7-848d-1a059d3fee13","Type":"ContainerStarted","Data":"e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd"}
Sep 16 19:35:32.139667 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:32.139564    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" event={"ID":"10681b13-4dc2-40b7-848d-1a059d3fee13","Type":"ContainerStarted","Data":"e79c46a4a045393bf6059e9a87621210d96e1208242c42eb01bc1bcef217878b"}
Sep 16 19:35:32.139667 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:32.139652    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:35:32.177203 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:32.177146    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" podStartSLOduration=1.177124272 podStartE2EDuration="1.177124272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:32.177022276 +0000 UTC m=+3347.701481112" watchObservedRunningTime="2026-09-16 19:35:32.177124272 +0000 UTC m=+3347.701583118"
Sep 16 19:35:34.150912 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:34.150802    2590 generic.go:356] "Generic (PLEG): container finished" podID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerID="8101331f166b25871bfa305bb45ac0ced24f4fb08ef36279ea0aa203bdeb785d" exitCode=0
Sep 16 19:35:34.150912 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:34.150850    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" event={"ID":"53e9a0cf-bb50-4fd0-9495-03e803d0cddc","Type":"ContainerDied","Data":"8101331f166b25871bfa305bb45ac0ced24f4fb08ef36279ea0aa203bdeb785d"}
Sep 16 19:35:35.157605 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:35.157567    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" event={"ID":"53e9a0cf-bb50-4fd0-9495-03e803d0cddc","Type":"ContainerStarted","Data":"73d4f64326cb1c25b5e66be116464ec62baa43fbcb068b2e3f847e086c8f0a96"}
Sep 16 19:35:35.186458 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:35.186394    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podStartSLOduration=2.970201657 podStartE2EDuration="5.186368201s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.216166544s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:35.182917421 +0000 UTC m=+3350.707376256" watchObservedRunningTime="2026-09-16 19:35:35.186368201 +0000 UTC m=+3350.710827037"
Sep 16 19:35:35.276432 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:35.276374    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:36.369077 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.369044    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"]
Sep 16 19:35:36.372452 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.372425    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.375156 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.375129    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:35:36.375305 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.375278    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:35:36.375481 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.375341    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-72bgx\""
Sep 16 19:35:36.375667 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.375647    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:35:36.375792 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.375771    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:35:36.376076 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.376059    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:35:36.389197 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.389174    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"]
Sep 16 19:35:36.412918 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.412890    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f71db56c-2ec8-4df0-a767-8bc5c8026736-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.413132 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.412933    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71db56c-2ec8-4df0-a767-8bc5c8026736-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.413132 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.412982    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l99rv\" (UniqueName: \"kubernetes.io/projected/f71db56c-2ec8-4df0-a767-8bc5c8026736-kube-api-access-l99rv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.413132 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.413027    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.413132 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.413048    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.413132 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.413073    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.413649 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.413172    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.413649 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.413214    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.415545 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.415521    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:36.513950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.513903    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f71db56c-2ec8-4df0-a767-8bc5c8026736-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514139 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.513967    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71db56c-2ec8-4df0-a767-8bc5c8026736-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514139 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514014    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99rv\" (UniqueName: \"kubernetes.io/projected/f71db56c-2ec8-4df0-a767-8bc5c8026736-kube-api-access-l99rv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514139 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514096    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514320 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514135    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514320 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514186    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514320 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514242    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514320 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514272    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514682 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514652    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f71db56c-2ec8-4df0-a767-8bc5c8026736-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514763 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514654    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514763 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514714    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514763 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514745    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.514929 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.514909    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.516926 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.516903    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71db56c-2ec8-4df0-a767-8bc5c8026736-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.517234 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.517212    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.524068 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.524023    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99rv\" (UniqueName: \"kubernetes.io/projected/f71db56c-2ec8-4df0-a767-8bc5c8026736-kube-api-access-l99rv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.667845 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.667759    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:36.684727 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.684702    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:36.975236 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:36.975205    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"]
Sep 16 19:35:36.975946 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:35:36.975915    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71db56c_2ec8_4df0_a767_8bc5c8026736.slice/crio-d4d88e24b94694ee687340b2b631b06834b201df0309ecd9fe3d530be1f5adf8 WatchSource:0}: Error finding container d4d88e24b94694ee687340b2b631b06834b201df0309ecd9fe3d530be1f5adf8: Status 404 returned error can't find the container with id d4d88e24b94694ee687340b2b631b06834b201df0309ecd9fe3d530be1f5adf8
Sep 16 19:35:37.169658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:37.169618    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" event={"ID":"f71db56c-2ec8-4df0-a767-8bc5c8026736","Type":"ContainerStarted","Data":"e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b"}
Sep 16 19:35:37.169847 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:37.169685    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" event={"ID":"f71db56c-2ec8-4df0-a767-8bc5c8026736","Type":"ContainerStarted","Data":"d4d88e24b94694ee687340b2b631b06834b201df0309ecd9fe3d530be1f5adf8"}
Sep 16 19:35:40.186134 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:40.186095    2590 generic.go:356] "Generic (PLEG): container finished" podID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerID="e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b" exitCode=0
Sep 16 19:35:40.186520 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:40.186180    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" event={"ID":"f71db56c-2ec8-4df0-a767-8bc5c8026736","Type":"ContainerDied","Data":"e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b"}
Sep 16 19:35:41.193626 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:41.193581    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" event={"ID":"f71db56c-2ec8-4df0-a767-8bc5c8026736","Type":"ContainerStarted","Data":"102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568"}
Sep 16 19:35:41.218245 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:41.218174    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podStartSLOduration=2.746483892 podStartE2EDuration="5.218151099s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.471667207s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:41.216222475 +0000 UTC m=+3356.740681305" watchObservedRunningTime="2026-09-16 19:35:41.218151099 +0000 UTC m=+3356.742609936"
Sep 16 19:35:41.276304 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:41.276268    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:35:41.277853 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:41.277826    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:35:43.035023 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:43.034987    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:44.449689 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:44.449652    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:46.685174 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:46.685127    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:46.685635 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:46.685259    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:35:46.686775 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:46.686742    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:35:48.372404 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:48.372365    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:50.502773 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:50.502740    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:51.277424 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:51.277380    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:35:54.380605 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:54.380568    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:56.685777 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:35:56.685711    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:36:01.277166 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:01.277125    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:36:03.146660 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:03.146625    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:36:06.686122 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:06.686076    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:36:10.129929 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:10.129895    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:11.211688 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:11.211645    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:11.276783 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:11.276721    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:36:11.279372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:11.279341    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:16.686205 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:16.686158    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:36:21.277198 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:21.277155    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:36:26.685648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:26.685597    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:36:31.276597 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:31.276541    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:36:36.686238 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:36.686132    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:36:41.277256 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:41.277206    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:36:46.685550 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:46.685507    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:36:51.277278 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:51.277232    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:36:56.685493 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:36:56.685453    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:37:01.277090 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:01.277043    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:06.685666 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:06.685620    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:37:11.276833 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:11.276792    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:13.954363 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:13.954310    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:16.685375 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:16.685336    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:37:21.277201 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:21.277156    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:26.685409 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:26.685368    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:37:31.277295 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:31.277252    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:36.685655 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:36.685610    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:37:41.277172 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:41.277125    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:46.685642 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:46.685596    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:37:51.277143 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:51.277106    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:56.686230 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:37:56.686177    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:38:01.277433 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:01.277374    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:38:06.685663 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:06.685567    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:38:11.286663 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:11.286633    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:38:11.294369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:11.294343    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:38:13.274196 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:13.274146    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:13.279185 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:13.279154    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"]
Sep 16 19:38:13.286658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:13.286626    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"]
Sep 16 19:38:14.537737 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:14.537700    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:16.695238 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:16.695206    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:38:16.703290 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:16.703261    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:38:23.146675 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:23.146635    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" podUID="10681b13-4dc2-40b7-848d-1a059d3fee13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:25.915671 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:25.915624    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:25.951398 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:25.951357    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"]
Sep 16 19:38:25.986174 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:25.986140    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:27.824140 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:27.824102    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:28.304534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:28.304493    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main" containerID="cri-o://73d4f64326cb1c25b5e66be116464ec62baa43fbcb068b2e3f847e086c8f0a96" gracePeriod=45
Sep 16 19:38:28.310223 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:28.310190    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" podUID="10681b13-4dc2-40b7-848d-1a059d3fee13" containerName="main" containerID="cri-o://e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd" gracePeriod=45
Sep 16 19:38:29.010634 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.010597    2590 generic.go:356] "Generic (PLEG): container finished" podID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerID="73d4f64326cb1c25b5e66be116464ec62baa43fbcb068b2e3f847e086c8f0a96" exitCode=0
Sep 16 19:38:29.011013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.010665    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" event={"ID":"53e9a0cf-bb50-4fd0-9495-03e803d0cddc","Type":"ContainerDied","Data":"73d4f64326cb1c25b5e66be116464ec62baa43fbcb068b2e3f847e086c8f0a96"}
Sep 16 19:38:29.048927 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.048897    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:38:29.059748 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.059715    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-dshm\") pod \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") "
Sep 16 19:38:29.059905 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.059762    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-cabundle-cert\") pod \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") "
Sep 16 19:38:29.059905 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.059785    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-home\") pod \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") "
Sep 16 19:38:29.059905 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.059810    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tls-certs\") pod \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") "
Sep 16 19:38:29.059905 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.059836    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvpvd\" (UniqueName: \"kubernetes.io/projected/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kube-api-access-mvpvd\") pod \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") "
Sep 16 19:38:29.059905 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.059852    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-model-cache\") pod \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") "
Sep 16 19:38:29.060298 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.060176    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53e9a0cf-bb50-4fd0-9495-03e803d0cddc" (UID: "53e9a0cf-bb50-4fd0-9495-03e803d0cddc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:38:29.060378 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.060254    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-model-cache" (OuterVolumeSpecName: "model-cache") pod "53e9a0cf-bb50-4fd0-9495-03e803d0cddc" (UID: "53e9a0cf-bb50-4fd0-9495-03e803d0cddc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:29.060499 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.060457    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-home" (OuterVolumeSpecName: "home") pod "53e9a0cf-bb50-4fd0-9495-03e803d0cddc" (UID: "53e9a0cf-bb50-4fd0-9495-03e803d0cddc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:29.061976 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.061948    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-dshm" (OuterVolumeSpecName: "dshm") pod "53e9a0cf-bb50-4fd0-9495-03e803d0cddc" (UID: "53e9a0cf-bb50-4fd0-9495-03e803d0cddc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:29.062154 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.062132    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53e9a0cf-bb50-4fd0-9495-03e803d0cddc" (UID: "53e9a0cf-bb50-4fd0-9495-03e803d0cddc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:29.062223 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.062204    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kube-api-access-mvpvd" (OuterVolumeSpecName: "kube-api-access-mvpvd") pod "53e9a0cf-bb50-4fd0-9495-03e803d0cddc" (UID: "53e9a0cf-bb50-4fd0-9495-03e803d0cddc"). InnerVolumeSpecName "kube-api-access-mvpvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:29.161312 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.161210    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kserve-provision-location\") pod \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") "
Sep 16 19:38:29.161312 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.161269    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tmp-dir\") pod \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\" (UID: \"53e9a0cf-bb50-4fd0-9495-03e803d0cddc\") "
Sep 16 19:38:29.161526 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.161504    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:29.161565 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.161529    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:29.161599 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.161572    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:29.161599 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.161586    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvpvd\" (UniqueName: \"kubernetes.io/projected/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kube-api-access-mvpvd\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:29.161662 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.161599    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:29.161662 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.161612    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:29.174472 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.174425    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53e9a0cf-bb50-4fd0-9495-03e803d0cddc" (UID: "53e9a0cf-bb50-4fd0-9495-03e803d0cddc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:29.185215 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.185180    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53e9a0cf-bb50-4fd0-9495-03e803d0cddc" (UID: "53e9a0cf-bb50-4fd0-9495-03e803d0cddc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:29.262612 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.262571    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:29.262612 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:29.262605    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e9a0cf-bb50-4fd0-9495-03e803d0cddc-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:30.016844 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:30.016804    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6" event={"ID":"53e9a0cf-bb50-4fd0-9495-03e803d0cddc","Type":"ContainerDied","Data":"f4fd10b9014e9b366b28a2118c7fd1dbd0f1cdc0feb5f53c21c7e25698b391ef"}
Sep 16 19:38:30.016844 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:30.016829    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"
Sep 16 19:38:30.017370 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:30.016875    2590 scope.go:117] "RemoveContainer" containerID="73d4f64326cb1c25b5e66be116464ec62baa43fbcb068b2e3f847e086c8f0a96"
Sep 16 19:38:30.029295 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:30.029271    2590 scope.go:117] "RemoveContainer" containerID="8101331f166b25871bfa305bb45ac0ced24f4fb08ef36279ea0aa203bdeb785d"
Sep 16 19:38:30.041451 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:30.041421    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"]
Sep 16 19:38:30.045616 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:30.045584    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-58d658465-n8jv6"]
Sep 16 19:38:31.025796 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:31.025749    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" path="/var/lib/kubelet/pods/53e9a0cf-bb50-4fd0-9495-03e803d0cddc/volumes"
Sep 16 19:38:33.146730 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:33.146689    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" podUID="10681b13-4dc2-40b7-848d-1a059d3fee13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:40.974216 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:40.974150    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main" containerID="cri-o://102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568" gracePeriod=45
Sep 16 19:38:41.716518 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.716494    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:38:41.770329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770246    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-tmp-dir\") pod \"f71db56c-2ec8-4df0-a767-8bc5c8026736\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") "
Sep 16 19:38:41.770329 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770303    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-model-cache\") pod \"f71db56c-2ec8-4df0-a767-8bc5c8026736\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") "
Sep 16 19:38:41.770552 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770355    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71db56c-2ec8-4df0-a767-8bc5c8026736-tls-certs\") pod \"f71db56c-2ec8-4df0-a767-8bc5c8026736\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") "
Sep 16 19:38:41.770552 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770379    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-dshm\") pod \"f71db56c-2ec8-4df0-a767-8bc5c8026736\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") "
Sep 16 19:38:41.770552 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770436    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f71db56c-2ec8-4df0-a767-8bc5c8026736-cabundle-cert\") pod \"f71db56c-2ec8-4df0-a767-8bc5c8026736\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") "
Sep 16 19:38:41.770714 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770562    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-kserve-provision-location\") pod \"f71db56c-2ec8-4df0-a767-8bc5c8026736\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") "
Sep 16 19:38:41.770714 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770638    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-home\") pod \"f71db56c-2ec8-4df0-a767-8bc5c8026736\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") "
Sep 16 19:38:41.770714 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770665    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l99rv\" (UniqueName: \"kubernetes.io/projected/f71db56c-2ec8-4df0-a767-8bc5c8026736-kube-api-access-l99rv\") pod \"f71db56c-2ec8-4df0-a767-8bc5c8026736\" (UID: \"f71db56c-2ec8-4df0-a767-8bc5c8026736\") "
Sep 16 19:38:41.770907 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770560    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-model-cache" (OuterVolumeSpecName: "model-cache") pod "f71db56c-2ec8-4df0-a767-8bc5c8026736" (UID: "f71db56c-2ec8-4df0-a767-8bc5c8026736"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:41.770907 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770791    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71db56c-2ec8-4df0-a767-8bc5c8026736-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f71db56c-2ec8-4df0-a767-8bc5c8026736" (UID: "f71db56c-2ec8-4df0-a767-8bc5c8026736"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:38:41.771015 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770950    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:41.771015 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.770969    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f71db56c-2ec8-4df0-a767-8bc5c8026736-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:41.771405 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.771369    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-home" (OuterVolumeSpecName: "home") pod "f71db56c-2ec8-4df0-a767-8bc5c8026736" (UID: "f71db56c-2ec8-4df0-a767-8bc5c8026736"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:41.773023 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.772991    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71db56c-2ec8-4df0-a767-8bc5c8026736-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f71db56c-2ec8-4df0-a767-8bc5c8026736" (UID: "f71db56c-2ec8-4df0-a767-8bc5c8026736"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:41.773125 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.773108    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-dshm" (OuterVolumeSpecName: "dshm") pod "f71db56c-2ec8-4df0-a767-8bc5c8026736" (UID: "f71db56c-2ec8-4df0-a767-8bc5c8026736"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:41.773474 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.773449    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71db56c-2ec8-4df0-a767-8bc5c8026736-kube-api-access-l99rv" (OuterVolumeSpecName: "kube-api-access-l99rv") pod "f71db56c-2ec8-4df0-a767-8bc5c8026736" (UID: "f71db56c-2ec8-4df0-a767-8bc5c8026736"). InnerVolumeSpecName "kube-api-access-l99rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:41.790323 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.790293    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f71db56c-2ec8-4df0-a767-8bc5c8026736" (UID: "f71db56c-2ec8-4df0-a767-8bc5c8026736"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:41.794215 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.794194    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f71db56c-2ec8-4df0-a767-8bc5c8026736" (UID: "f71db56c-2ec8-4df0-a767-8bc5c8026736"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:41.871866 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.871824    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:41.871866 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.871857    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71db56c-2ec8-4df0-a767-8bc5c8026736-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:41.871866 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.871869    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:41.872137 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.871907    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:41.872137 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.871922    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71db56c-2ec8-4df0-a767-8bc5c8026736-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:41.872137 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:41.871933    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l99rv\" (UniqueName: \"kubernetes.io/projected/f71db56c-2ec8-4df0-a767-8bc5c8026736-kube-api-access-l99rv\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:42.075827 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.075374    2590 generic.go:356] "Generic (PLEG): container finished" podID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerID="102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568" exitCode=0
Sep 16 19:38:42.075827 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.075518    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" event={"ID":"f71db56c-2ec8-4df0-a767-8bc5c8026736","Type":"ContainerDied","Data":"102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568"}
Sep 16 19:38:42.075827 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.075609    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq" event={"ID":"f71db56c-2ec8-4df0-a767-8bc5c8026736","Type":"ContainerDied","Data":"d4d88e24b94694ee687340b2b631b06834b201df0309ecd9fe3d530be1f5adf8"}
Sep 16 19:38:42.075827 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.075647    2590 scope.go:117] "RemoveContainer" containerID="102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568"
Sep 16 19:38:42.076390 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.075829    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"
Sep 16 19:38:42.084861 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.084844    2590 scope.go:117] "RemoveContainer" containerID="e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b"
Sep 16 19:38:42.092980 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.092942    2590 scope.go:117] "RemoveContainer" containerID="102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568"
Sep 16 19:38:42.093243 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:38:42.093223    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568\": container with ID starting with 102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568 not found: ID does not exist" containerID="102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568"
Sep 16 19:38:42.093333 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.093252    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568"} err="failed to get container status \"102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568\": rpc error: code = NotFound desc = could not find container \"102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568\": container with ID starting with 102cb089121ce82871b488af79b184c363dbe2b9bdeb8c8cffdd6efff2ef1568 not found: ID does not exist"
Sep 16 19:38:42.093333 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.093270    2590 scope.go:117] "RemoveContainer" containerID="e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b"
Sep 16 19:38:42.093542 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:38:42.093524    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b\": container with ID starting with e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b not found: ID does not exist" containerID="e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b"
Sep 16 19:38:42.093587 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.093550    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b"} err="failed to get container status \"e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b\": rpc error: code = NotFound desc = could not find container \"e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b\": container with ID starting with e767f34a80e1115b64deaafa677e15e624e1d16dcb99e69b96a2286f4f3ce48b not found: ID does not exist"
Sep 16 19:38:42.099862 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.099836    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"]
Sep 16 19:38:42.105270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:42.105241    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85vwhhq"]
Sep 16 19:38:43.025477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:43.025433    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" path="/var/lib/kubelet/pods/f71db56c-2ec8-4df0-a767-8bc5c8026736/volumes"
Sep 16 19:38:43.146822 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:43.146777    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" podUID="10681b13-4dc2-40b7-848d-1a059d3fee13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:43.147241 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:43.146917    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:38:53.147064 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:53.147020    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" podUID="10681b13-4dc2-40b7-848d-1a059d3fee13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:58.560579 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:58.560552    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:38:58.708448 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:58.708341    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10681b13-4dc2-40b7-848d-1a059d3fee13-tls-certs\") pod \"10681b13-4dc2-40b7-848d-1a059d3fee13\" (UID: \"10681b13-4dc2-40b7-848d-1a059d3fee13\") "
Sep 16 19:38:58.708618 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:58.708473    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhqxb\" (UniqueName: \"kubernetes.io/projected/10681b13-4dc2-40b7-848d-1a059d3fee13-kube-api-access-rhqxb\") pod \"10681b13-4dc2-40b7-848d-1a059d3fee13\" (UID: \"10681b13-4dc2-40b7-848d-1a059d3fee13\") "
Sep 16 19:38:58.710847 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:58.710811    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10681b13-4dc2-40b7-848d-1a059d3fee13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10681b13-4dc2-40b7-848d-1a059d3fee13" (UID: "10681b13-4dc2-40b7-848d-1a059d3fee13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:58.710982 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:58.710812    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10681b13-4dc2-40b7-848d-1a059d3fee13-kube-api-access-rhqxb" (OuterVolumeSpecName: "kube-api-access-rhqxb") pod "10681b13-4dc2-40b7-848d-1a059d3fee13" (UID: "10681b13-4dc2-40b7-848d-1a059d3fee13"). InnerVolumeSpecName "kube-api-access-rhqxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:58.809703 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:58.809667    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhqxb\" (UniqueName: \"kubernetes.io/projected/10681b13-4dc2-40b7-848d-1a059d3fee13-kube-api-access-rhqxb\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:58.809703 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:58.809699    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10681b13-4dc2-40b7-848d-1a059d3fee13-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:38:59.152235 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:59.152198    2590 generic.go:356] "Generic (PLEG): container finished" podID="10681b13-4dc2-40b7-848d-1a059d3fee13" containerID="e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd" exitCode=0
Sep 16 19:38:59.152414 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:59.152244    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"
Sep 16 19:38:59.152414 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:59.152283    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" event={"ID":"10681b13-4dc2-40b7-848d-1a059d3fee13","Type":"ContainerDied","Data":"e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd"}
Sep 16 19:38:59.152414 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:59.152337    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b" event={"ID":"10681b13-4dc2-40b7-848d-1a059d3fee13","Type":"ContainerDied","Data":"e79c46a4a045393bf6059e9a87621210d96e1208242c42eb01bc1bcef217878b"}
Sep 16 19:38:59.152414 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:59.152370    2590 scope.go:117] "RemoveContainer" containerID="e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd"
Sep 16 19:38:59.160417 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:59.160397    2590 scope.go:117] "RemoveContainer" containerID="e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd"
Sep 16 19:38:59.160732 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:38:59.160709    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd\": container with ID starting with e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd not found: ID does not exist" containerID="e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd"
Sep 16 19:38:59.160786 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:59.160744    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd"} err="failed to get container status \"e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd\": rpc error: code = NotFound desc = could not find container \"e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd\": container with ID starting with e41e1d363591b95325c109659e42b8f2bb2a1c9188a6d8857be09f07f20c2fbd not found: ID does not exist"
Sep 16 19:38:59.168931 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:59.168904    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"]
Sep 16 19:38:59.172985 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:38:59.172965    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5cr5b"]
Sep 16 19:39:01.025273 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:01.025239    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10681b13-4dc2-40b7-848d-1a059d3fee13" path="/var/lib/kubelet/pods/10681b13-4dc2-40b7-848d-1a059d3fee13/volumes"
Sep 16 19:39:02.818035 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:02.818003    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:02.921934 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:02.921902    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:03.024676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:03.024625    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:09.469561 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:09.469528    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:13.362648 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.362614    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"]
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.362923    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.362938    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.362948    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="storage-initializer"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.362953    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="storage-initializer"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.362975    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10681b13-4dc2-40b7-848d-1a059d3fee13" containerName="main"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.362980    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="10681b13-4dc2-40b7-848d-1a059d3fee13" containerName="main"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.362989    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="storage-initializer"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.362995    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="storage-initializer"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.363001    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.363006    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.363053    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71db56c-2ec8-4df0-a767-8bc5c8026736" containerName="main"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.363061    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="10681b13-4dc2-40b7-848d-1a059d3fee13" containerName="main"
Sep 16 19:39:13.363116 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.363067    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="53e9a0cf-bb50-4fd0-9495-03e803d0cddc" containerName="main"
Sep 16 19:39:13.367445 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.367424    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.370209 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.370183    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 16 19:39:13.370341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.370183    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 16 19:39:13.370341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.370251    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-s2sbv\""
Sep 16 19:39:13.370341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.370287    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 16 19:39:13.370341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.370194    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 16 19:39:13.370341 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.370327    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:39:13.379459 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.379436    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"]
Sep 16 19:39:13.397571 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.397546    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:13.532665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.532619    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-home\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.532665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.532665    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f74q\" (UniqueName: \"kubernetes.io/projected/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kube-api-access-9f74q\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.532950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.532695    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tmp-dir\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.532950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.532711    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.532950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.532731    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-model-cache\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.532950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.532745    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tls-certs\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.532950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.532763    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecd9977-65de-4566-9a94-9f22cc8cd55d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.532950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.532777    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-dshm\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634077 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.633995    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-home\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634077 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634033    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f74q\" (UniqueName: \"kubernetes.io/projected/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kube-api-access-9f74q\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634077 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634063    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tmp-dir\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634362 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634080    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634362 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634110    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-model-cache\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634362 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634249    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tls-certs\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634362 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634296    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecd9977-65de-4566-9a94-9f22cc8cd55d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634362 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634326    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-dshm\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634575 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634399    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-home\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634575 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634453    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tmp-dir\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634575 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634517    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634575 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634545    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-model-cache\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.634943 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.634923    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecd9977-65de-4566-9a94-9f22cc8cd55d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.636638 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.636620    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-dshm\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.637020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.637003    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tls-certs\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.645160 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.645131    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f74q\" (UniqueName: \"kubernetes.io/projected/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kube-api-access-9f74q\") pod \"lora-single-adapter-test-kserve-db6b86697-n47dn\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.678141 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.678096    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:13.966243 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:13.966205    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"]
Sep 16 19:39:14.219611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:14.219493    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" event={"ID":"3ecd9977-65de-4566-9a94-9f22cc8cd55d","Type":"ContainerStarted","Data":"a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469"}
Sep 16 19:39:14.219611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:14.219571    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" event={"ID":"3ecd9977-65de-4566-9a94-9f22cc8cd55d","Type":"ContainerStarted","Data":"8858f01ca06f35a8426553f1b58de03d2427229b4b86c540c99c104976ca3b33"}
Sep 16 19:39:15.835667 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:15.835618    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:15.911087 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:15.911037    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:15.991567 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:15.991538    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:16.095997 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:16.095914    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:16.174472 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:16.174422    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:18.245080 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:18.245042    2590 generic.go:356] "Generic (PLEG): container finished" podID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerID="a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469" exitCode=0
Sep 16 19:39:18.245498 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:18.245106    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" event={"ID":"3ecd9977-65de-4566-9a94-9f22cc8cd55d","Type":"ContainerDied","Data":"a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469"}
Sep 16 19:39:19.253424 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:19.253384    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" event={"ID":"3ecd9977-65de-4566-9a94-9f22cc8cd55d","Type":"ContainerStarted","Data":"1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263"}
Sep 16 19:39:19.279901 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:19.279830    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podStartSLOduration=3.035155383 podStartE2EDuration="6.27981251s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.244657127s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:19.277861436 +0000 UTC m=+3574.802320271" watchObservedRunningTime="2026-09-16 19:39:19.27981251 +0000 UTC m=+3574.804271344"
Sep 16 19:39:19.678990 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:19.678937    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:22.697618 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:22.697589    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:23.678900 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:23.678835    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:39:23.680364 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:23.680325    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 16 19:39:28.833251 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:28.833215    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:28.931059 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:28.931024    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:33.678738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:33.678691    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 16 19:39:37.145524 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.145445    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"]
Sep 16 19:39:37.148668 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.148645    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.153486 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.153460    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:39:37.153649 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.153457    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:39:37.153649 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.153566    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-jpvfk\""
Sep 16 19:39:37.153649 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.153583    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:39:37.154172 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.154143    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:39:37.154309 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.154189    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:39:37.175757 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.175711    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"]
Sep 16 19:39:37.234099 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.234064    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e905f3-a770-42b0-b608-d9803094a6f9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.234310 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.234116    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9rl\" (UniqueName: \"kubernetes.io/projected/75e905f3-a770-42b0-b608-d9803094a6f9-kube-api-access-rs9rl\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.234310 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.234145    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.234310 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.234197    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.234310 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.234228    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e905f3-a770-42b0-b608-d9803094a6f9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.234310 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.234263    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.234516 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.234315    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.234516 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.234347    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335128 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335092    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335298 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335157    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335298 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335191    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e905f3-a770-42b0-b608-d9803094a6f9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335298 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335251    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9rl\" (UniqueName: \"kubernetes.io/projected/75e905f3-a770-42b0-b608-d9803094a6f9-kube-api-access-rs9rl\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335466 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335293    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335466 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335322    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335466 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335361    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e905f3-a770-42b0-b608-d9803094a6f9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335466 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335411    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335708 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335564    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335708 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335581    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.335708 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335663    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.336014 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.335985    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.336287 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.336265    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e905f3-a770-42b0-b608-d9803094a6f9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.337836 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.337810    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.338240 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.338215    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e905f3-a770-42b0-b608-d9803094a6f9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.353126 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.353097    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9rl\" (UniqueName: \"kubernetes.io/projected/75e905f3-a770-42b0-b608-d9803094a6f9-kube-api-access-rs9rl\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.392197 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.392160    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:37.459688 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.459596    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:37.629943 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.629909    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:37.757378 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:37.757291    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"]
Sep 16 19:39:38.342265 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:38.342218    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" event={"ID":"75e905f3-a770-42b0-b608-d9803094a6f9","Type":"ContainerStarted","Data":"5b376c5a06408337844b98ed270b651067940e33caeadc954eb4e65b88a13359"}
Sep 16 19:39:38.342715 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:38.342292    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" event={"ID":"75e905f3-a770-42b0-b608-d9803094a6f9","Type":"ContainerStarted","Data":"ad13a21bb86d3b53acfa950b3146e2b5f60cba06e1ec2a80fa383182a3f19835"}
Sep 16 19:39:40.352900 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:40.352840    2590 generic.go:356] "Generic (PLEG): container finished" podID="75e905f3-a770-42b0-b608-d9803094a6f9" containerID="5b376c5a06408337844b98ed270b651067940e33caeadc954eb4e65b88a13359" exitCode=0
Sep 16 19:39:40.353439 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:40.352918    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" event={"ID":"75e905f3-a770-42b0-b608-d9803094a6f9","Type":"ContainerDied","Data":"5b376c5a06408337844b98ed270b651067940e33caeadc954eb4e65b88a13359"}
Sep 16 19:39:41.360962 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:41.360923    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" event={"ID":"75e905f3-a770-42b0-b608-d9803094a6f9","Type":"ContainerStarted","Data":"d0e482bb67c1f49a87115e43dd09282db7b04548e6922cb52e5c62b313b15b72"}
Sep 16 19:39:41.392167 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:41.392091    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podStartSLOduration=1.9571279700000002 podStartE2EDuration="4.392066493s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.434938523s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:41.390955509 +0000 UTC m=+3596.915414350" watchObservedRunningTime="2026-09-16 19:39:41.392066493 +0000 UTC m=+3596.916525328"
Sep 16 19:39:43.679013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:43.678957    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 16 19:39:47.460125 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:47.460083    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:47.460573 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:47.460156    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:39:47.461689 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:47.461661    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:39:53.679595 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:53.679538    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 16 19:39:57.460182 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:39:57.460137    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:40:03.678870 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:03.678822    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 16 19:40:07.460426 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:07.460376    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:40:08.175643 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:08.175604    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:13.678660 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:13.678610    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 16 19:40:17.461027 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:17.460967    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:40:23.678818 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:23.678774    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 16 19:40:27.460588 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:27.460545    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:40:33.679330 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:33.679278    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 16 19:40:37.460692 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:37.460643    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:40:43.688489 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:43.688458    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:40:43.696298 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:43.696271    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:40:47.460688 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:47.460646    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:40:57.460996 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:40:57.460948    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:41:07.460717 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:07.460620    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:41:17.460536 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:17.460482    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:41:24.776162 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:24.776114    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-db6b86697-n47dn_3ecd9977-65de-4566-9a94-9f22cc8cd55d/storage-initializer/0.log"
Sep 16 19:41:24.784982 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:24.784955    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-db6b86697-n47dn_3ecd9977-65de-4566-9a94-9f22cc8cd55d/main/0.log"
Sep 16 19:41:26.393514 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:26.393476    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"]
Sep 16 19:41:26.399025 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:26.398982    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:27.460295 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:27.460250    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:41:28.735616 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:28.735577    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:37.460305 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:37.460262    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:41:41.420966 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:41.420898    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" containerID="cri-o://1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263" gracePeriod=45
Sep 16 19:41:41.689607 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:41.689504    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 16 19:41:42.379921 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.379870    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:41:42.563676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.563635    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kserve-provision-location\") pod \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") "
Sep 16 19:41:42.563676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.563674    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-home\") pod \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") "
Sep 16 19:41:42.564275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.563702    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f74q\" (UniqueName: \"kubernetes.io/projected/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kube-api-access-9f74q\") pod \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") "
Sep 16 19:41:42.564275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.563744    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tls-certs\") pod \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") "
Sep 16 19:41:42.564275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.563772    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-dshm\") pod \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") "
Sep 16 19:41:42.564275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.563820    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-model-cache\") pod \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") "
Sep 16 19:41:42.564275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.563910    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tmp-dir\") pod \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") "
Sep 16 19:41:42.564275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.563938    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecd9977-65de-4566-9a94-9f22cc8cd55d-cabundle-cert\") pod \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\" (UID: \"3ecd9977-65de-4566-9a94-9f22cc8cd55d\") "
Sep 16 19:41:42.564275 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.564161    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-home" (OuterVolumeSpecName: "home") pod "3ecd9977-65de-4566-9a94-9f22cc8cd55d" (UID: "3ecd9977-65de-4566-9a94-9f22cc8cd55d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:42.564823 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.564474    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ecd9977-65de-4566-9a94-9f22cc8cd55d" (UID: "3ecd9977-65de-4566-9a94-9f22cc8cd55d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:42.564823 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.564483    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecd9977-65de-4566-9a94-9f22cc8cd55d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ecd9977-65de-4566-9a94-9f22cc8cd55d" (UID: "3ecd9977-65de-4566-9a94-9f22cc8cd55d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:41:42.566477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.566445    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kube-api-access-9f74q" (OuterVolumeSpecName: "kube-api-access-9f74q") pod "3ecd9977-65de-4566-9a94-9f22cc8cd55d" (UID: "3ecd9977-65de-4566-9a94-9f22cc8cd55d"). InnerVolumeSpecName "kube-api-access-9f74q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:41:42.566719 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.566700    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-dshm" (OuterVolumeSpecName: "dshm") pod "3ecd9977-65de-4566-9a94-9f22cc8cd55d" (UID: "3ecd9977-65de-4566-9a94-9f22cc8cd55d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:42.566790 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.566731    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ecd9977-65de-4566-9a94-9f22cc8cd55d" (UID: "3ecd9977-65de-4566-9a94-9f22cc8cd55d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:41:42.577001 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.576973    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ecd9977-65de-4566-9a94-9f22cc8cd55d" (UID: "3ecd9977-65de-4566-9a94-9f22cc8cd55d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:42.599016 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.598969    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ecd9977-65de-4566-9a94-9f22cc8cd55d" (UID: "3ecd9977-65de-4566-9a94-9f22cc8cd55d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:42.664775 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.664740    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f74q\" (UniqueName: \"kubernetes.io/projected/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kube-api-access-9f74q\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:41:42.664775 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.664770    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:41:42.664775 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.664780    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:41:42.665052 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.664789    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:41:42.665052 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.664823    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:41:42.665052 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.664831    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecd9977-65de-4566-9a94-9f22cc8cd55d-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:41:42.665052 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.664841    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:41:42.665052 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.664849    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ecd9977-65de-4566-9a94-9f22cc8cd55d-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:41:42.967698 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.967601    2590 generic.go:356] "Generic (PLEG): container finished" podID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerID="1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263" exitCode=0
Sep 16 19:41:42.967698 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.967652    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"
Sep 16 19:41:42.967698 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.967677    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" event={"ID":"3ecd9977-65de-4566-9a94-9f22cc8cd55d","Type":"ContainerDied","Data":"1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263"}
Sep 16 19:41:42.967968 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.967752    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn" event={"ID":"3ecd9977-65de-4566-9a94-9f22cc8cd55d","Type":"ContainerDied","Data":"8858f01ca06f35a8426553f1b58de03d2427229b4b86c540c99c104976ca3b33"}
Sep 16 19:41:42.967968 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.967775    2590 scope.go:117] "RemoveContainer" containerID="1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263"
Sep 16 19:41:42.977346 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.977329    2590 scope.go:117] "RemoveContainer" containerID="a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469"
Sep 16 19:41:42.985466 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.985445    2590 scope.go:117] "RemoveContainer" containerID="1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263"
Sep 16 19:41:42.985783 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:41:42.985755    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263\": container with ID starting with 1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263 not found: ID does not exist" containerID="1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263"
Sep 16 19:41:42.985856 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.985791    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263"} err="failed to get container status \"1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263\": rpc error: code = NotFound desc = could not find container \"1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263\": container with ID starting with 1ce71106747719223e2d48778150555265216e44eabd4d199be9fddcd8007263 not found: ID does not exist"
Sep 16 19:41:42.985856 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.985809    2590 scope.go:117] "RemoveContainer" containerID="a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469"
Sep 16 19:41:42.986124 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:41:42.986106    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469\": container with ID starting with a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469 not found: ID does not exist" containerID="a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469"
Sep 16 19:41:42.986187 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.986129    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469"} err="failed to get container status \"a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469\": rpc error: code = NotFound desc = could not find container \"a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469\": container with ID starting with a42bc4247239140c7125f434242ff6f0f5d3a689dcc8ccfe12b85ff7c5290469 not found: ID does not exist"
Sep 16 19:41:42.991929 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.991907    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"]
Sep 16 19:41:42.997025 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:42.997003    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-db6b86697-n47dn"]
Sep 16 19:41:43.025521 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:43.025482    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" path="/var/lib/kubelet/pods/3ecd9977-65de-4566-9a94-9f22cc8cd55d/volumes"
Sep 16 19:41:46.497092 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:46.497032    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:46.602744 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:46.602710    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:46.709196 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:46.709156    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:47.460461 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:47.460420    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:41:52.776923 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:52.776860    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:52.875963 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:52.875926    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:57.470673 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:57.470641    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:41:57.478633 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:41:57.478602    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:42:03.946076 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.946042    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"]
Sep 16 19:42:03.946493 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.946344    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="storage-initializer"
Sep 16 19:42:03.946493 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.946356    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="storage-initializer"
Sep 16 19:42:03.946493 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.946386    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main"
Sep 16 19:42:03.946493 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.946391    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main"
Sep 16 19:42:03.946493 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.946440    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ecd9977-65de-4566-9a94-9f22cc8cd55d" containerName="main"
Sep 16 19:42:03.949268 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.949251    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:03.951845 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.951798    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:42:03.951845 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.951836    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 16 19:42:03.952092 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.951853    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-tttx5\""
Sep 16 19:42:03.952092 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.951873    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 16 19:42:03.952179 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.952144    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 16 19:42:03.952179 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.952147    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 16 19:42:03.966750 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.966715    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"]
Sep 16 19:42:03.970251 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:03.970219    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:04.044074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.044030    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.044281 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.044080    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.044281 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.044111    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9qw\" (UniqueName: \"kubernetes.io/projected/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kube-api-access-4t9qw\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.044281 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.044152    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-model-cache\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.044281 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.044225    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-home\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.044281 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.044254    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.044498 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.044306    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-dshm\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.044498 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.044323    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tls-certs\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145048 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145003    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-model-cache\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145057    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-home\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145082    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145131    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-dshm\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145152    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tls-certs\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145184    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145217    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145263 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145243    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9qw\" (UniqueName: \"kubernetes.io/projected/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kube-api-access-4t9qw\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145609 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145492    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-home\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145609 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145547    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145716 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145682    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.145807 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.145788    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-model-cache\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.146078 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.146060    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.147803 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.147780    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-dshm\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.147972 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.147954    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tls-certs\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.155150 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.155113    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9qw\" (UniqueName: \"kubernetes.io/projected/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kube-api-access-4t9qw\") pod \"lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.259349 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.259263    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:04.545663 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:04.545627    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"]
Sep 16 19:42:04.549065 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:42:04.549030    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a8a3f0_21de_4467_99a9_7f3b45b479cf.slice/crio-0badfa62578832c19aa8dc1621ca3542d1f76c53ab5c079ccf45e006c1b0adb4 WatchSource:0}: Error finding container 0badfa62578832c19aa8dc1621ca3542d1f76c53ab5c079ccf45e006c1b0adb4: Status 404 returned error can't find the container with id 0badfa62578832c19aa8dc1621ca3542d1f76c53ab5c079ccf45e006c1b0adb4
Sep 16 19:42:05.071372 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:05.071339    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" event={"ID":"c7a8a3f0-21de-4467-99a9-7f3b45b479cf","Type":"ContainerStarted","Data":"42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11"}
Sep 16 19:42:05.071857 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:05.071392    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" event={"ID":"c7a8a3f0-21de-4467-99a9-7f3b45b479cf","Type":"ContainerStarted","Data":"0badfa62578832c19aa8dc1621ca3542d1f76c53ab5c079ccf45e006c1b0adb4"}
Sep 16 19:42:08.086364 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:08.086328    2590 generic.go:356] "Generic (PLEG): container finished" podID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerID="42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11" exitCode=0
Sep 16 19:42:08.086831 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:08.086406    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" event={"ID":"c7a8a3f0-21de-4467-99a9-7f3b45b479cf","Type":"ContainerDied","Data":"42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11"}
Sep 16 19:42:09.092526 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:09.092489    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" event={"ID":"c7a8a3f0-21de-4467-99a9-7f3b45b479cf","Type":"ContainerStarted","Data":"fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8"}
Sep 16 19:42:09.119528 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:09.119461    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podStartSLOduration=3.042449043 podStartE2EDuration="6.119439332s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.076990289s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:09.118252691 +0000 UTC m=+3744.642711537" watchObservedRunningTime="2026-09-16 19:42:09.119439332 +0000 UTC m=+3744.643898169"
Sep 16 19:42:09.259834 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:09.259785    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:14.260384 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:14.260344    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:42:14.262128 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:14.262094    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:42:22.498224 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:22.498181    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:22.505688 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:22.505656    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"]
Sep 16 19:42:22.575980 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:22.575948    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:24.259913 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:24.259839    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:42:25.413267 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:25.413228    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:25.473030 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:25.472979    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:34.259945 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:34.259865    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:42:37.529975 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:37.529861    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main" containerID="cri-o://d0e482bb67c1f49a87115e43dd09282db7b04548e6922cb52e5c62b313b15b72" gracePeriod=45
Sep 16 19:42:38.233244 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.233204    2590 generic.go:356] "Generic (PLEG): container finished" podID="75e905f3-a770-42b0-b608-d9803094a6f9" containerID="d0e482bb67c1f49a87115e43dd09282db7b04548e6922cb52e5c62b313b15b72" exitCode=0
Sep 16 19:42:38.233406 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.233275    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" event={"ID":"75e905f3-a770-42b0-b608-d9803094a6f9","Type":"ContainerDied","Data":"d0e482bb67c1f49a87115e43dd09282db7b04548e6922cb52e5c62b313b15b72"}
Sep 16 19:42:38.271100 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.271074    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:42:38.353202 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353115    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-kserve-provision-location\") pod \"75e905f3-a770-42b0-b608-d9803094a6f9\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") "
Sep 16 19:42:38.353202 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353164    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-dshm\") pod \"75e905f3-a770-42b0-b608-d9803094a6f9\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") "
Sep 16 19:42:38.353202 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353188    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e905f3-a770-42b0-b608-d9803094a6f9-tls-certs\") pod \"75e905f3-a770-42b0-b608-d9803094a6f9\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") "
Sep 16 19:42:38.353490 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353223    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-tmp-dir\") pod \"75e905f3-a770-42b0-b608-d9803094a6f9\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") "
Sep 16 19:42:38.353490 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353337    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e905f3-a770-42b0-b608-d9803094a6f9-cabundle-cert\") pod \"75e905f3-a770-42b0-b608-d9803094a6f9\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") "
Sep 16 19:42:38.353490 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353384    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-model-cache\") pod \"75e905f3-a770-42b0-b608-d9803094a6f9\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") "
Sep 16 19:42:38.353490 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353411    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs9rl\" (UniqueName: \"kubernetes.io/projected/75e905f3-a770-42b0-b608-d9803094a6f9-kube-api-access-rs9rl\") pod \"75e905f3-a770-42b0-b608-d9803094a6f9\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") "
Sep 16 19:42:38.353490 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353438    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-home\") pod \"75e905f3-a770-42b0-b608-d9803094a6f9\" (UID: \"75e905f3-a770-42b0-b608-d9803094a6f9\") "
Sep 16 19:42:38.354090 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353853    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "75e905f3-a770-42b0-b608-d9803094a6f9" (UID: "75e905f3-a770-42b0-b608-d9803094a6f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.354090 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.353928    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e905f3-a770-42b0-b608-d9803094a6f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75e905f3-a770-42b0-b608-d9803094a6f9" (UID: "75e905f3-a770-42b0-b608-d9803094a6f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:42:38.354404 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.354362    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-home" (OuterVolumeSpecName: "home") pod "75e905f3-a770-42b0-b608-d9803094a6f9" (UID: "75e905f3-a770-42b0-b608-d9803094a6f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.356030 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.356000    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-dshm" (OuterVolumeSpecName: "dshm") pod "75e905f3-a770-42b0-b608-d9803094a6f9" (UID: "75e905f3-a770-42b0-b608-d9803094a6f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.356172 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.356154    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e905f3-a770-42b0-b608-d9803094a6f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75e905f3-a770-42b0-b608-d9803094a6f9" (UID: "75e905f3-a770-42b0-b608-d9803094a6f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:38.356517 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.356487    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e905f3-a770-42b0-b608-d9803094a6f9-kube-api-access-rs9rl" (OuterVolumeSpecName: "kube-api-access-rs9rl") pod "75e905f3-a770-42b0-b608-d9803094a6f9" (UID: "75e905f3-a770-42b0-b608-d9803094a6f9"). InnerVolumeSpecName "kube-api-access-rs9rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:38.372737 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.372699    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75e905f3-a770-42b0-b608-d9803094a6f9" (UID: "75e905f3-a770-42b0-b608-d9803094a6f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.379895 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.379843    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75e905f3-a770-42b0-b608-d9803094a6f9" (UID: "75e905f3-a770-42b0-b608-d9803094a6f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.454498 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.454461    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e905f3-a770-42b0-b608-d9803094a6f9-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.454498 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.454494    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.454729 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.454507    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9rl\" (UniqueName: \"kubernetes.io/projected/75e905f3-a770-42b0-b608-d9803094a6f9-kube-api-access-rs9rl\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.454729 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.454519    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.454729 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.454532    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.454729 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.454544    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.454729 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.454554    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e905f3-a770-42b0-b608-d9803094a6f9-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.454729 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:38.454566    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e905f3-a770-42b0-b608-d9803094a6f9-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:42:39.241047 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:39.241003    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl" event={"ID":"75e905f3-a770-42b0-b608-d9803094a6f9","Type":"ContainerDied","Data":"ad13a21bb86d3b53acfa950b3146e2b5f60cba06e1ec2a80fa383182a3f19835"}
Sep 16 19:42:39.241047 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:39.241046    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"
Sep 16 19:42:39.241521 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:39.241087    2590 scope.go:117] "RemoveContainer" containerID="d0e482bb67c1f49a87115e43dd09282db7b04548e6922cb52e5c62b313b15b72"
Sep 16 19:42:39.249458 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:39.249434    2590 scope.go:117] "RemoveContainer" containerID="5b376c5a06408337844b98ed270b651067940e33caeadc954eb4e65b88a13359"
Sep 16 19:42:39.264668 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:39.264642    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"]
Sep 16 19:42:39.268986 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:39.268964    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bfb7894db-hvvbl"]
Sep 16 19:42:41.026095 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:41.026059    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" path="/var/lib/kubelet/pods/75e905f3-a770-42b0-b608-d9803094a6f9/volumes"
Sep 16 19:42:44.260234 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:44.260181    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:42:54.260045 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:42:54.259993    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:43:04.260082 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:04.260027    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:43:11.627203 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.627160    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:11.735845 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.735808    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:11.964158 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.964125    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"]
Sep 16 19:43:11.964578 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.964558    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="storage-initializer"
Sep 16 19:43:11.964626 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.964585    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="storage-initializer"
Sep 16 19:43:11.964626 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.964598    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main"
Sep 16 19:43:11.964626 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.964606    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main"
Sep 16 19:43:11.964718 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.964694    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e905f3-a770-42b0-b608-d9803094a6f9" containerName="main"
Sep 16 19:43:11.968008 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.967989    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:11.971632 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.971607    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:43:11.971632 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.971622    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-dklz7\""
Sep 16 19:43:11.971852 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.971645    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:43:11.972835 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.972819    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:43:11.989304 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:11.989280    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"]
Sep 16 19:43:12.002536 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.002511    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:12.041216 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.041180    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.041397 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.041224    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.041397 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.041242    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.041397 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.041264    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e82897-eb31-487e-97f5-147b9fdc1c35-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.041397 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.041291    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.041397 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.041312    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.041397 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.041339    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.041669 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.041445    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.041669 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.041502    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgzc\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-kube-api-access-lqgzc\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.090095 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.090064    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:12.142865 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.142824    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.142907    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.142944    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgzc\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-kube-api-access-lqgzc\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.142962    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.142999    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.143021    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.143044    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e82897-eb31-487e-97f5-147b9fdc1c35-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143074 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.143070    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143446 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.143100    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143446 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.143370    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143446 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.143381    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.143867 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.143838    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e82897-eb31-487e-97f5-147b9fdc1c35-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.144187 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.144167    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.144265 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.144243    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.145578 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.145551    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.145821 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.145799    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.152894 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.152854    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgzc\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-kube-api-access-lqgzc\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.154185 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.154166    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-n6j4v\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.278262 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.278227    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:12.570348 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.570260    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"]
Sep 16 19:43:12.573804 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:43:12.573765    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e82897_eb31_487e_97f5_147b9fdc1c35.slice/crio-973b630777a597741b20d71b3877b98e9c05a3c534e384f9f56df46b3debd03a WatchSource:0}: Error finding container 973b630777a597741b20d71b3877b98e9c05a3c534e384f9f56df46b3debd03a: Status 404 returned error can't find the container with id 973b630777a597741b20d71b3877b98e9c05a3c534e384f9f56df46b3debd03a
Sep 16 19:43:12.575952 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.575919    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:43:12.576043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.575987    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:43:12.576043 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:12.576014    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:43:13.399912 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:13.399862    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" event={"ID":"45e82897-eb31-487e-97f5-147b9fdc1c35","Type":"ContainerStarted","Data":"414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413"}
Sep 16 19:43:13.400388 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:13.399930    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" event={"ID":"45e82897-eb31-487e-97f5-147b9fdc1c35","Type":"ContainerStarted","Data":"973b630777a597741b20d71b3877b98e9c05a3c534e384f9f56df46b3debd03a"}
Sep 16 19:43:13.430481 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:13.430426    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" podStartSLOduration=2.43040892 podStartE2EDuration="2.43040892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:13.427536014 +0000 UTC m=+3808.951994847" watchObservedRunningTime="2026-09-16 19:43:13.43040892 +0000 UTC m=+3808.954867754"
Sep 16 19:43:14.259978 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:14.259928    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:43:14.279375 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:14.279337    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:14.284455 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:14.284428    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:14.405681 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:14.405646    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:14.406553 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:14.406528    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:15.750089 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:15.750054    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:18.503027 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.502994    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"]
Sep 16 19:43:18.506490 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.506472    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.509189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.509155    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:43:18.509404 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.509386    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:43:18.509499 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.509386    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:43:18.509805 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.509782    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-94r2g\""
Sep 16 19:43:18.521138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.521095    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:18.524898 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.521366    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"]
Sep 16 19:43:18.595361 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.595323    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:18.597461 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.597435    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.597614 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.597471    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.597614 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.597494    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.597614 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.597516    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154b049b-2058-450f-b9f5-598d024e071f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.597614 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.597597    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgcd\" (UniqueName: \"kubernetes.io/projected/154b049b-2058-450f-b9f5-598d024e071f-kube-api-access-vfgcd\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.597802 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.597652    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-home\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.597802 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.597681    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/154b049b-2058-450f-b9f5-598d024e071f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.597802 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.597709    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699162 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699117    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699171    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699219    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699252    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154b049b-2058-450f-b9f5-598d024e071f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699299    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgcd\" (UniqueName: \"kubernetes.io/projected/154b049b-2058-450f-b9f5-598d024e071f-kube-api-access-vfgcd\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699344    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-home\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699367    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/154b049b-2058-450f-b9f5-598d024e071f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699710 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699404    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699710 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699602    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699710 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699662    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699901 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699758    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.699969 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.699939    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-home\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.700207 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.700189    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/154b049b-2058-450f-b9f5-598d024e071f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.702238 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.702212    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.702542 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.702522    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154b049b-2058-450f-b9f5-598d024e071f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.707670 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.707646    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgcd\" (UniqueName: \"kubernetes.io/projected/154b049b-2058-450f-b9f5-598d024e071f-kube-api-access-vfgcd\") pod \"router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:18.831079 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:18.831035    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:19.126989 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:19.126866    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"]
Sep 16 19:43:19.146849 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:19.146819    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:19.148152 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:43:19.148122    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154b049b_2058_450f_b9f5_598d024e071f.slice/crio-d0eb988b2019c8bb09a72a26bceae94444b9f609391d7b6ebaee8a63a7a5ee86 WatchSource:0}: Error finding container d0eb988b2019c8bb09a72a26bceae94444b9f609391d7b6ebaee8a63a7a5ee86: Status 404 returned error can't find the container with id d0eb988b2019c8bb09a72a26bceae94444b9f609391d7b6ebaee8a63a7a5ee86
Sep 16 19:43:19.247902 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:19.247827    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:19.436699 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:19.436583    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" event={"ID":"154b049b-2058-450f-b9f5-598d024e071f","Type":"ContainerStarted","Data":"d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11"}
Sep 16 19:43:19.436699 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:19.436656    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" event={"ID":"154b049b-2058-450f-b9f5-598d024e071f","Type":"ContainerStarted","Data":"d0eb988b2019c8bb09a72a26bceae94444b9f609391d7b6ebaee8a63a7a5ee86"}
Sep 16 19:43:19.834002 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:19.833958    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:22.457802 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:22.457746    2590 generic.go:356] "Generic (PLEG): container finished" podID="154b049b-2058-450f-b9f5-598d024e071f" containerID="d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11" exitCode=0
Sep 16 19:43:22.458331 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:22.457817    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" event={"ID":"154b049b-2058-450f-b9f5-598d024e071f","Type":"ContainerDied","Data":"d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11"}
Sep 16 19:43:23.465376 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:23.465345    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" event={"ID":"154b049b-2058-450f-b9f5-598d024e071f","Type":"ContainerStarted","Data":"7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93"}
Sep 16 19:43:23.495531 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:23.495477    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podStartSLOduration=2.700331382 podStartE2EDuration="5.495459224s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.795127842s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:23.495044147 +0000 UTC m=+3819.019502983" watchObservedRunningTime="2026-09-16 19:43:23.495459224 +0000 UTC m=+3819.019918058"
Sep 16 19:43:24.260312 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:24.260260    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:43:24.472254 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:24.472216    2590 generic.go:356] "Generic (PLEG): container finished" podID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerID="414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413" exitCode=0
Sep 16 19:43:24.472254 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:24.472258    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" event={"ID":"45e82897-eb31-487e-97f5-147b9fdc1c35","Type":"ContainerDied","Data":"414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413"}
Sep 16 19:43:24.472840 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:24.472639    2590 scope.go:117] "RemoveContainer" containerID="414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413"
Sep 16 19:43:24.473811 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:24.473766    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:43:24.473951 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:24.473872    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:43:24.473951 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:24.473943    2590 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:43:25.478559 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:25.478526    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" event={"ID":"45e82897-eb31-487e-97f5-147b9fdc1c35","Type":"ContainerStarted","Data":"7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638"}
Sep 16 19:43:25.863920 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:25.863856    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:25.933083 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:25.933043    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:26.278646 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:26.278611    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:26.279944 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:26.279910    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 19:43:27.279122 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:27.279072    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 19:43:28.279024 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:28.278978    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 19:43:28.834268 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:28.834232    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:28.834868 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:28.834846    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:43:28.835749 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:28.835718    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:43:29.279452 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:29.279416    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 19:43:29.406199 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:29.406145    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:30.156744 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:30.156705    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:30.278900 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:30.278843    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 19:43:31.283185 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:31.283152    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:31.284417 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:31.284395    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:43:34.270642 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:34.270607    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:43:34.529247 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:34.529217    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:43:38.832080 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:38.832030    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:43:48.831722 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:48.831671    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:43:58.832152 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:43:58.832101    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:44:04.190496 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:04.190451    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq_c7a8a3f0-21de-4467-99a9-7f3b45b479cf/storage-initializer/0.log"
Sep 16 19:44:04.207349 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:04.207321    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq_c7a8a3f0-21de-4467-99a9-7f3b45b479cf/main/0.log"
Sep 16 19:44:05.468966 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:05.468931    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"]
Sep 16 19:44:05.483041 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:05.483002    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:08.832057 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:08.832011    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:44:18.832003 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:18.831949    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:44:20.491793 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:20.491728    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main" containerID="cri-o://fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8" gracePeriod=45
Sep 16 19:44:21.147642 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.147616    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:44:21.270872 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.270788    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tls-certs\") pod \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") "
Sep 16 19:44:21.270872 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.270853    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-model-cache\") pod \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") "
Sep 16 19:44:21.271085 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.270905    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t9qw\" (UniqueName: \"kubernetes.io/projected/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kube-api-access-4t9qw\") pod \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") "
Sep 16 19:44:21.271138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.271078    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-home\") pod \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") "
Sep 16 19:44:21.271138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.271118    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-cabundle-cert\") pod \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") "
Sep 16 19:44:21.271248 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.271151    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kserve-provision-location\") pod \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") "
Sep 16 19:44:21.271248 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.271200    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tmp-dir\") pod \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") "
Sep 16 19:44:21.271248 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.271206    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7a8a3f0-21de-4467-99a9-7f3b45b479cf" (UID: "c7a8a3f0-21de-4467-99a9-7f3b45b479cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:21.271248 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.271236    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-dshm\") pod \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\" (UID: \"c7a8a3f0-21de-4467-99a9-7f3b45b479cf\") "
Sep 16 19:44:21.271554 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.271496    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-home" (OuterVolumeSpecName: "home") pod "c7a8a3f0-21de-4467-99a9-7f3b45b479cf" (UID: "c7a8a3f0-21de-4467-99a9-7f3b45b479cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:21.271554 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.271506    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7a8a3f0-21de-4467-99a9-7f3b45b479cf" (UID: "c7a8a3f0-21de-4467-99a9-7f3b45b479cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:44:21.271554 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.271511    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:44:21.273811 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.273778    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-dshm" (OuterVolumeSpecName: "dshm") pod "c7a8a3f0-21de-4467-99a9-7f3b45b479cf" (UID: "c7a8a3f0-21de-4467-99a9-7f3b45b479cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:21.273811 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.273778    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7a8a3f0-21de-4467-99a9-7f3b45b479cf" (UID: "c7a8a3f0-21de-4467-99a9-7f3b45b479cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:44:21.274027 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.273820    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kube-api-access-4t9qw" (OuterVolumeSpecName: "kube-api-access-4t9qw") pod "c7a8a3f0-21de-4467-99a9-7f3b45b479cf" (UID: "c7a8a3f0-21de-4467-99a9-7f3b45b479cf"). InnerVolumeSpecName "kube-api-access-4t9qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:44:21.288182 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.288141    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7a8a3f0-21de-4467-99a9-7f3b45b479cf" (UID: "c7a8a3f0-21de-4467-99a9-7f3b45b479cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:21.298184 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.298155    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7a8a3f0-21de-4467-99a9-7f3b45b479cf" (UID: "c7a8a3f0-21de-4467-99a9-7f3b45b479cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:21.372477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.372433    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:44:21.372477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.372478    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:44:21.372693 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.372496    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9qw\" (UniqueName: \"kubernetes.io/projected/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kube-api-access-4t9qw\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:44:21.372693 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.372513    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:44:21.372693 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.372525    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:44:21.372693 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.372541    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:44:21.372693 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.372554    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a8a3f0-21de-4467-99a9-7f3b45b479cf-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:44:21.767081 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.767043    2590 generic.go:356] "Generic (PLEG): container finished" podID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerID="fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8" exitCode=0
Sep 16 19:44:21.767584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.767101    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" event={"ID":"c7a8a3f0-21de-4467-99a9-7f3b45b479cf","Type":"ContainerDied","Data":"fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8"}
Sep 16 19:44:21.767584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.767135    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"
Sep 16 19:44:21.767584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.767178    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq" event={"ID":"c7a8a3f0-21de-4467-99a9-7f3b45b479cf","Type":"ContainerDied","Data":"0badfa62578832c19aa8dc1621ca3542d1f76c53ab5c079ccf45e006c1b0adb4"}
Sep 16 19:44:21.767584 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.767213    2590 scope.go:117] "RemoveContainer" containerID="fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8"
Sep 16 19:44:21.776710 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.776689    2590 scope.go:117] "RemoveContainer" containerID="42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11"
Sep 16 19:44:21.785681 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.785664    2590 scope.go:117] "RemoveContainer" containerID="fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8"
Sep 16 19:44:21.785984 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:44:21.785959    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8\": container with ID starting with fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8 not found: ID does not exist" containerID="fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8"
Sep 16 19:44:21.786094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.785991    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8"} err="failed to get container status \"fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8\": rpc error: code = NotFound desc = could not find container \"fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8\": container with ID starting with fc8ad31d66b679e33e9051830f8df83cf93e0f2c0fbeaae8fcb6fef18d7dbcb8 not found: ID does not exist"
Sep 16 19:44:21.786094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.786010    2590 scope.go:117] "RemoveContainer" containerID="42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11"
Sep 16 19:44:21.786344 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:44:21.786320    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11\": container with ID starting with 42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11 not found: ID does not exist" containerID="42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11"
Sep 16 19:44:21.786391 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.786355    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11"} err="failed to get container status \"42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11\": rpc error: code = NotFound desc = could not find container \"42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11\": container with ID starting with 42d37e1b7a1e70edde9cd2f26cecc8d5ea87faf0f43e6488ec484d9306f8fd11 not found: ID does not exist"
Sep 16 19:44:21.798356 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.798327    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"]
Sep 16 19:44:21.807692 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:21.807664    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-569d7bb55b-hxgqq"]
Sep 16 19:44:23.026162 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:23.026116    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" path="/var/lib/kubelet/pods/c7a8a3f0-21de-4467-99a9-7f3b45b479cf/volumes"
Sep 16 19:44:25.580897 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:25.580851    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:25.680684 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:25.680651    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:25.787014 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:25.786980    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:28.831592 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:28.831547    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:44:31.896517 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:31.896477    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:31.996572 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:31.996539    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:34.380858 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.380821    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"]
Sep 16 19:44:34.381272 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.381115    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="storage-initializer"
Sep 16 19:44:34.381272 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.381126    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="storage-initializer"
Sep 16 19:44:34.381272 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.381147    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main"
Sep 16 19:44:34.381272 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.381153    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main"
Sep 16 19:44:34.381272 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.381209    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a8a3f0-21de-4467-99a9-7f3b45b479cf" containerName="main"
Sep 16 19:44:34.383750 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.383730    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.389617 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.389584    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:44:34.389758 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.389669    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 19:44:34.389978 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.389959    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 16 19:44:34.392464 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.392443    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 19:44:34.392593 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.392551    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-x2khv\""
Sep 16 19:44:34.392987 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.392954    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 19:44:34.403212 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.402735    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"]
Sep 16 19:44:34.454356 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.454326    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:34.480740 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.480697    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43b70f1a-a78e-444b-ab89-118249089726-cabundle-cert\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.480740 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.480737    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-dshm\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.480964 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.480756    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzc2n\" (UniqueName: \"kubernetes.io/projected/43b70f1a-a78e-444b-ab89-118249089726-kube-api-access-zzc2n\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.480964 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.480782    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-home\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.480964 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.480823    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-kserve-provision-location\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.480964 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.480895    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-model-cache\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.480964 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.480948    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70f1a-a78e-444b-ab89-118249089726-tls-certs\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.481127 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.480990    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-tmp-dir\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.581727 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.581679    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43b70f1a-a78e-444b-ab89-118249089726-cabundle-cert\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.581727 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.581725    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-dshm\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582026 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.581750    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzc2n\" (UniqueName: \"kubernetes.io/projected/43b70f1a-a78e-444b-ab89-118249089726-kube-api-access-zzc2n\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582026 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.581784    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-home\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582026 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.581814    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-kserve-provision-location\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582026 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.581837    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-model-cache\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582026 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.581862    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70f1a-a78e-444b-ab89-118249089726-tls-certs\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582026 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.581917    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-tmp-dir\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582533 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.582504    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-model-cache\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582717 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.582521    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-tmp-dir\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582717 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.582566    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-home\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582947 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.582636    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-kserve-provision-location\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.582947 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.582775    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43b70f1a-a78e-444b-ab89-118249089726-cabundle-cert\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.585106 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.585079    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-dshm\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.585313 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.585295    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70f1a-a78e-444b-ab89-118249089726-tls-certs\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.608154 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.608112    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzc2n\" (UniqueName: \"kubernetes.io/projected/43b70f1a-a78e-444b-ab89-118249089726-kube-api-access-zzc2n\") pod \"tls-verification-test-kserve-64b9dd657b-kdz7x\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.692683 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.692604    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"]
Sep 16 19:44:34.696304 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.696287    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:44:34.700464 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.700439    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:34.718240 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.718206    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-m9g8x\""
Sep 16 19:44:34.759044 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.758904    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"]
Sep 16 19:44:34.771389 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.771350    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:34.783749 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.783711    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8pn8\" (UniqueName: \"kubernetes.io/projected/aaabae5a-c1e7-4f41-a483-087846bd2825-kube-api-access-h8pn8\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq\" (UID: \"aaabae5a-c1e7-4f41-a483-087846bd2825\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:44:34.783955 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.783865    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaabae5a-c1e7-4f41-a483-087846bd2825-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq\" (UID: \"aaabae5a-c1e7-4f41-a483-087846bd2825\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:44:34.885014 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.884975    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaabae5a-c1e7-4f41-a483-087846bd2825-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq\" (UID: \"aaabae5a-c1e7-4f41-a483-087846bd2825\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:44:34.885227 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.885046    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8pn8\" (UniqueName: \"kubernetes.io/projected/aaabae5a-c1e7-4f41-a483-087846bd2825-kube-api-access-h8pn8\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq\" (UID: \"aaabae5a-c1e7-4f41-a483-087846bd2825\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:44:34.887930 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.887907    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaabae5a-c1e7-4f41-a483-087846bd2825-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq\" (UID: \"aaabae5a-c1e7-4f41-a483-087846bd2825\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:44:34.894864 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.894839    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8pn8\" (UniqueName: \"kubernetes.io/projected/aaabae5a-c1e7-4f41-a483-087846bd2825-kube-api-access-h8pn8\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq\" (UID: \"aaabae5a-c1e7-4f41-a483-087846bd2825\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:44:34.983236 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:34.983207    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"]
Sep 16 19:44:34.985825 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:44:34.985785    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b70f1a_a78e_444b_ab89_118249089726.slice/crio-d7be42c61a15034d8d6f6f1fa45d77bd8dc7fd9cc6f111d5f37b237eee555b24 WatchSource:0}: Error finding container d7be42c61a15034d8d6f6f1fa45d77bd8dc7fd9cc6f111d5f37b237eee555b24: Status 404 returned error can't find the container with id d7be42c61a15034d8d6f6f1fa45d77bd8dc7fd9cc6f111d5f37b237eee555b24
Sep 16 19:44:35.006696 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:35.006672    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:44:35.293507 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:44:35.293459    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaabae5a_c1e7_4f41_a483_087846bd2825.slice/crio-e2fa462073360df8929e1ab5b165d018bfe8c119105ce22308b7c9bd9691eba8 WatchSource:0}: Error finding container e2fa462073360df8929e1ab5b165d018bfe8c119105ce22308b7c9bd9691eba8: Status 404 returned error can't find the container with id e2fa462073360df8929e1ab5b165d018bfe8c119105ce22308b7c9bd9691eba8
Sep 16 19:44:35.294477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:35.294335    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"]
Sep 16 19:44:35.836966 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:35.836922    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" event={"ID":"aaabae5a-c1e7-4f41-a483-087846bd2825","Type":"ContainerStarted","Data":"b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c"}
Sep 16 19:44:35.837498 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:35.836995    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" event={"ID":"aaabae5a-c1e7-4f41-a483-087846bd2825","Type":"ContainerStarted","Data":"e2fa462073360df8929e1ab5b165d018bfe8c119105ce22308b7c9bd9691eba8"}
Sep 16 19:44:35.837498 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:35.837028    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:44:35.838733 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:35.838705    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" event={"ID":"43b70f1a-a78e-444b-ab89-118249089726","Type":"ContainerStarted","Data":"f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7"}
Sep 16 19:44:35.838853 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:35.838747    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" event={"ID":"43b70f1a-a78e-444b-ab89-118249089726","Type":"ContainerStarted","Data":"d7be42c61a15034d8d6f6f1fa45d77bd8dc7fd9cc6f111d5f37b237eee555b24"}
Sep 16 19:44:35.856840 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:35.856776    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" podStartSLOduration=1.856758191 podStartE2EDuration="1.856758191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:44:35.854748724 +0000 UTC m=+3891.379207557" watchObservedRunningTime="2026-09-16 19:44:35.856758191 +0000 UTC m=+3891.381217021"
Sep 16 19:44:37.857940 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:37.857904    2590 generic.go:356] "Generic (PLEG): container finished" podID="43b70f1a-a78e-444b-ab89-118249089726" containerID="f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7" exitCode=0
Sep 16 19:44:37.858320 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:37.857978    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" event={"ID":"43b70f1a-a78e-444b-ab89-118249089726","Type":"ContainerDied","Data":"f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7"}
Sep 16 19:44:38.832362 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:38.832314    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:44:38.866957 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:38.866912    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" event={"ID":"43b70f1a-a78e-444b-ab89-118249089726","Type":"ContainerStarted","Data":"eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2"}
Sep 16 19:44:38.903954 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:38.903861    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podStartSLOduration=2.337738035 podStartE2EDuration="4.903838287s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.566100252s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:44:38.894061969 +0000 UTC m=+3894.418520803" watchObservedRunningTime="2026-09-16 19:44:38.903838287 +0000 UTC m=+3894.428297122"
Sep 16 19:44:39.701182 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:39.701142    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:44.701630 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:44.701576    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:44:44.703156 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:44.703126    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:44:46.618260 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:46.618219    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:48.832015 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:48.831961    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:44:54.701231 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:54.701189    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:44:58.832033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:44:58.831976    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:45:04.701381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:04.701332    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:45:06.853823 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:06.853773    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:45:08.832133 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:08.832085    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:45:14.701348 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:14.701290    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:45:18.832145 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:18.832089    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:45:24.700899 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:24.700840    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:45:28.841809 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:28.841760    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:45:28.850309 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:28.850278    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:45:34.701576 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:34.701529    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:45:38.675427 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:38.675341    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:38.710603 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:38.710570    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"]
Sep 16 19:45:38.752387 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:38.752355    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:44.701968 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:44.701923    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:45:53.734234 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:53.734182    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main" containerID="cri-o://7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93" gracePeriod=45
Sep 16 19:45:54.701607 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.701565    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:45:54.782387 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.782358    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:45:54.910375 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.910281    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-dshm\") pod \"154b049b-2058-450f-b9f5-598d024e071f\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") "
Sep 16 19:45:54.910375 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.910367    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-tmp-dir\") pod \"154b049b-2058-450f-b9f5-598d024e071f\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") "
Sep 16 19:45:54.910622 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.910403    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-model-cache\") pod \"154b049b-2058-450f-b9f5-598d024e071f\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") "
Sep 16 19:45:54.910622 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.910444    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154b049b-2058-450f-b9f5-598d024e071f-tls-certs\") pod \"154b049b-2058-450f-b9f5-598d024e071f\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") "
Sep 16 19:45:54.910622 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.910470    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfgcd\" (UniqueName: \"kubernetes.io/projected/154b049b-2058-450f-b9f5-598d024e071f-kube-api-access-vfgcd\") pod \"154b049b-2058-450f-b9f5-598d024e071f\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") "
Sep 16 19:45:54.910622 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.910504    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/154b049b-2058-450f-b9f5-598d024e071f-cabundle-cert\") pod \"154b049b-2058-450f-b9f5-598d024e071f\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") "
Sep 16 19:45:54.910622 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.910537    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-home\") pod \"154b049b-2058-450f-b9f5-598d024e071f\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") "
Sep 16 19:45:54.910622 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.910572    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-kserve-provision-location\") pod \"154b049b-2058-450f-b9f5-598d024e071f\" (UID: \"154b049b-2058-450f-b9f5-598d024e071f\") "
Sep 16 19:45:54.911485 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.911455    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-home" (OuterVolumeSpecName: "home") pod "154b049b-2058-450f-b9f5-598d024e071f" (UID: "154b049b-2058-450f-b9f5-598d024e071f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:54.911598 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.911554    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154b049b-2058-450f-b9f5-598d024e071f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "154b049b-2058-450f-b9f5-598d024e071f" (UID: "154b049b-2058-450f-b9f5-598d024e071f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:45:54.912976 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.912944    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/154b049b-2058-450f-b9f5-598d024e071f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "154b049b-2058-450f-b9f5-598d024e071f" (UID: "154b049b-2058-450f-b9f5-598d024e071f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:54.913152 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.913127    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-dshm" (OuterVolumeSpecName: "dshm") pod "154b049b-2058-450f-b9f5-598d024e071f" (UID: "154b049b-2058-450f-b9f5-598d024e071f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:54.913445 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.913414    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-model-cache" (OuterVolumeSpecName: "model-cache") pod "154b049b-2058-450f-b9f5-598d024e071f" (UID: "154b049b-2058-450f-b9f5-598d024e071f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:54.913754 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.913733    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154b049b-2058-450f-b9f5-598d024e071f-kube-api-access-vfgcd" (OuterVolumeSpecName: "kube-api-access-vfgcd") pod "154b049b-2058-450f-b9f5-598d024e071f" (UID: "154b049b-2058-450f-b9f5-598d024e071f"). InnerVolumeSpecName "kube-api-access-vfgcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:54.929977 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.929932    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "154b049b-2058-450f-b9f5-598d024e071f" (UID: "154b049b-2058-450f-b9f5-598d024e071f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:54.939548 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:54.939507    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "154b049b-2058-450f-b9f5-598d024e071f" (UID: "154b049b-2058-450f-b9f5-598d024e071f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:55.011782 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.011746    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154b049b-2058-450f-b9f5-598d024e071f-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:45:55.011782 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.011774    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgcd\" (UniqueName: \"kubernetes.io/projected/154b049b-2058-450f-b9f5-598d024e071f-kube-api-access-vfgcd\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:45:55.011782 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.011787    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/154b049b-2058-450f-b9f5-598d024e071f-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:45:55.012144 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.011799    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:45:55.012144 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.011807    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:45:55.012144 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.011814    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:45:55.012144 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.011822    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:45:55.012144 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.011829    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154b049b-2058-450f-b9f5-598d024e071f-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:45:55.248519 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.248432    2590 generic.go:356] "Generic (PLEG): container finished" podID="154b049b-2058-450f-b9f5-598d024e071f" containerID="7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93" exitCode=0
Sep 16 19:45:55.248519 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.248471    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" event={"ID":"154b049b-2058-450f-b9f5-598d024e071f","Type":"ContainerDied","Data":"7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93"}
Sep 16 19:45:55.248519 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.248513    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69" event={"ID":"154b049b-2058-450f-b9f5-598d024e071f","Type":"ContainerDied","Data":"d0eb988b2019c8bb09a72a26bceae94444b9f609391d7b6ebaee8a63a7a5ee86"}
Sep 16 19:45:55.248737 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.248528    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"
Sep 16 19:45:55.248737 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.248535    2590 scope.go:117] "RemoveContainer" containerID="7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93"
Sep 16 19:45:55.257556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.257535    2590 scope.go:117] "RemoveContainer" containerID="d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11"
Sep 16 19:45:55.267069 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.266799    2590 scope.go:117] "RemoveContainer" containerID="7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93"
Sep 16 19:45:55.267275 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:45:55.267241    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93\": container with ID starting with 7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93 not found: ID does not exist" containerID="7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93"
Sep 16 19:45:55.267381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.267283    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93"} err="failed to get container status \"7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93\": rpc error: code = NotFound desc = could not find container \"7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93\": container with ID starting with 7d81f3238801edce717a1cf8f046020ad12114e32f30148e1e5548a646d18e93 not found: ID does not exist"
Sep 16 19:45:55.267381 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.267309    2590 scope.go:117] "RemoveContainer" containerID="d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11"
Sep 16 19:45:55.267697 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:45:55.267662    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11\": container with ID starting with d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11 not found: ID does not exist" containerID="d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11"
Sep 16 19:45:55.267766 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.267700    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11"} err="failed to get container status \"d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11\": rpc error: code = NotFound desc = could not find container \"d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11\": container with ID starting with d9145f7eb486d710d31b283b6e941671cd90b77c169dc5566b30086f50608c11 not found: ID does not exist"
Sep 16 19:45:55.268063 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.268046    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"]
Sep 16 19:45:55.274163 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:55.274142    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-bb469d858-l7v69"]
Sep 16 19:45:57.026950 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:45:57.026909    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="154b049b-2058-450f-b9f5-598d024e071f" path="/var/lib/kubelet/pods/154b049b-2058-450f-b9f5-598d024e071f/volumes"
Sep 16 19:46:04.702660 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:04.702612    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:46:14.701349 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:14.701305    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:46:24.701183 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:24.701137    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:46:27.766347 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:27.766307    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:28.811100 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:28.811062    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:28.900066 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:28.900031    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:33.756961 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:33.756923    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"]
Sep 16 19:46:33.757377 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:33.757144    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy" containerID="cri-o://7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638" gracePeriod=30
Sep 16 19:46:33.852364 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:33.852275    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:34.711167 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:34.711133    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:46:34.718747 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:34.718723    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:46:36.535609 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:36.535572    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:36.774522 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:36.774491    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:39.005253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.005225    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:46:39.100536 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100502    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-certs\") pod \"45e82897-eb31-487e-97f5-147b9fdc1c35\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") "
Sep 16 19:46:39.100684 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100568    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqgzc\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-kube-api-access-lqgzc\") pod \"45e82897-eb31-487e-97f5-147b9fdc1c35\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") "
Sep 16 19:46:39.100684 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100620    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e82897-eb31-487e-97f5-147b9fdc1c35-istiod-ca-cert\") pod \"45e82897-eb31-487e-97f5-147b9fdc1c35\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") "
Sep 16 19:46:39.100776 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100683    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-data\") pod \"45e82897-eb31-487e-97f5-147b9fdc1c35\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") "
Sep 16 19:46:39.100776 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100716    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-socket\") pod \"45e82897-eb31-487e-97f5-147b9fdc1c35\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") "
Sep 16 19:46:39.100776 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100747    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-envoy\") pod \"45e82897-eb31-487e-97f5-147b9fdc1c35\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") "
Sep 16 19:46:39.100776 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100745    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45e82897-eb31-487e-97f5-147b9fdc1c35" (UID: "45e82897-eb31-487e-97f5-147b9fdc1c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:39.101005 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100798    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-token\") pod \"45e82897-eb31-487e-97f5-147b9fdc1c35\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") "
Sep 16 19:46:39.101005 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100841    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-podinfo\") pod \"45e82897-eb31-487e-97f5-147b9fdc1c35\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") "
Sep 16 19:46:39.101005 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100869    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-credential-socket\") pod \"45e82897-eb31-487e-97f5-147b9fdc1c35\" (UID: \"45e82897-eb31-487e-97f5-147b9fdc1c35\") "
Sep 16 19:46:39.101160 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.100997    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-data" (OuterVolumeSpecName: "istio-data") pod "45e82897-eb31-487e-97f5-147b9fdc1c35" (UID: "45e82897-eb31-487e-97f5-147b9fdc1c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:39.101160 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.101051    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e82897-eb31-487e-97f5-147b9fdc1c35-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45e82897-eb31-487e-97f5-147b9fdc1c35" (UID: "45e82897-eb31-487e-97f5-147b9fdc1c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:46:39.101160 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.101091    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45e82897-eb31-487e-97f5-147b9fdc1c35" (UID: "45e82897-eb31-487e-97f5-147b9fdc1c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:39.101326 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.101190    2590 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-socket\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.101326 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.101209    2590 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-workload-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.101326 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.101224    2590 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e82897-eb31-487e-97f5-147b9fdc1c35-istiod-ca-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.101326 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.101236    2590 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-data\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.101572 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.101548    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45e82897-eb31-487e-97f5-147b9fdc1c35" (UID: "45e82897-eb31-487e-97f5-147b9fdc1c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:39.103206 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.103184    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-kube-api-access-lqgzc" (OuterVolumeSpecName: "kube-api-access-lqgzc") pod "45e82897-eb31-487e-97f5-147b9fdc1c35" (UID: "45e82897-eb31-487e-97f5-147b9fdc1c35"). InnerVolumeSpecName "kube-api-access-lqgzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:39.103336 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.103312    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45e82897-eb31-487e-97f5-147b9fdc1c35" (UID: "45e82897-eb31-487e-97f5-147b9fdc1c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:46:39.105232 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.103714    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45e82897-eb31-487e-97f5-147b9fdc1c35" (UID: "45e82897-eb31-487e-97f5-147b9fdc1c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:39.105554 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.105529    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-token" (OuterVolumeSpecName: "istio-token") pod "45e82897-eb31-487e-97f5-147b9fdc1c35" (UID: "45e82897-eb31-487e-97f5-147b9fdc1c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:39.201942 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.201873    2590 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-credential-socket\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.201942 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.201936    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqgzc\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-kube-api-access-lqgzc\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.201942 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.201946    2590 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-envoy\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.201942 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.201955    2590 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-token\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.202228 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.201964    2590 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e82897-eb31-487e-97f5-147b9fdc1c35-istio-podinfo\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.462171 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.462076    2590 generic.go:356] "Generic (PLEG): container finished" podID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerID="7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638" exitCode=0
Sep 16 19:46:39.462171 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.462129    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" event={"ID":"45e82897-eb31-487e-97f5-147b9fdc1c35","Type":"ContainerDied","Data":"7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638"}
Sep 16 19:46:39.462358 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.462183    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v" event={"ID":"45e82897-eb31-487e-97f5-147b9fdc1c35","Type":"ContainerDied","Data":"973b630777a597741b20d71b3877b98e9c05a3c534e384f9f56df46b3debd03a"}
Sep 16 19:46:39.462358 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.462131    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"
Sep 16 19:46:39.462358 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.462215    2590 scope.go:117] "RemoveContainer" containerID="7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638"
Sep 16 19:46:39.471998 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.471977    2590 scope.go:117] "RemoveContainer" containerID="414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413"
Sep 16 19:46:39.480493 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.480472    2590 scope.go:117] "RemoveContainer" containerID="7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638"
Sep 16 19:46:39.480768 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:46:39.480747    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638\": container with ID starting with 7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638 not found: ID does not exist" containerID="7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638"
Sep 16 19:46:39.480848 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.480783    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638"} err="failed to get container status \"7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638\": rpc error: code = NotFound desc = could not find container \"7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638\": container with ID starting with 7b79a0c0714b9f20835563b0debe2570d949377009d5a523c445e27244a04638 not found: ID does not exist"
Sep 16 19:46:39.480848 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.480810    2590 scope.go:117] "RemoveContainer" containerID="414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413"
Sep 16 19:46:39.481102 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:46:39.481080    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413\": container with ID starting with 414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413 not found: ID does not exist" containerID="414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413"
Sep 16 19:46:39.481145 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.481109    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413"} err="failed to get container status \"414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413\": rpc error: code = NotFound desc = could not find container \"414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413\": container with ID starting with 414a38539042edeed07dda38ca693b0a3d7ed0b3598250c3814d00e6977b2413 not found: ID does not exist"
Sep 16 19:46:39.487994 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.487971    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"]
Sep 16 19:46:39.491350 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:39.491325    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-n6j4v"]
Sep 16 19:46:41.026054 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:41.026017    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" path="/var/lib/kubelet/pods/45e82897-eb31-487e-97f5-147b9fdc1c35/volumes"
Sep 16 19:46:56.272060 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:56.272019    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:56.722231 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:56.722191    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:56.940134 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:56.940095    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:59.434578 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:59.434538    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:59.435142 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:59.434539    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"]
Sep 16 19:46:59.436334 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:59.436308    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"]
Sep 16 19:46:59.501676 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:46:59.501642    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:02.912290 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:02.912252    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:06.853486 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:06.853402    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" podUID="aaabae5a-c1e7-4f41-a483-087846bd2825" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:47:12.787611 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:12.787579    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:14.456292 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:14.456246    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" podUID="aaabae5a-c1e7-4f41-a483-087846bd2825" containerName="main" containerID="cri-o://b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c" gracePeriod=45
Sep 16 19:47:14.457665 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:14.457606    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" containerID="cri-o://eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2" gracePeriod=45
Sep 16 19:47:14.712700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:14.712606    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:47:15.211486 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.211461    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:47:15.311519 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.311480    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-home\") pod \"43b70f1a-a78e-444b-ab89-118249089726\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") "
Sep 16 19:47:15.311519 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.311523    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43b70f1a-a78e-444b-ab89-118249089726-cabundle-cert\") pod \"43b70f1a-a78e-444b-ab89-118249089726\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") "
Sep 16 19:47:15.311834 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.311548    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzc2n\" (UniqueName: \"kubernetes.io/projected/43b70f1a-a78e-444b-ab89-118249089726-kube-api-access-zzc2n\") pod \"43b70f1a-a78e-444b-ab89-118249089726\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") "
Sep 16 19:47:15.311834 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.311574    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-dshm\") pod \"43b70f1a-a78e-444b-ab89-118249089726\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") "
Sep 16 19:47:15.311834 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.311606    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70f1a-a78e-444b-ab89-118249089726-tls-certs\") pod \"43b70f1a-a78e-444b-ab89-118249089726\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") "
Sep 16 19:47:15.311834 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.311647    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-model-cache\") pod \"43b70f1a-a78e-444b-ab89-118249089726\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") "
Sep 16 19:47:15.311834 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.311705    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-kserve-provision-location\") pod \"43b70f1a-a78e-444b-ab89-118249089726\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") "
Sep 16 19:47:15.311834 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.311750    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-tmp-dir\") pod \"43b70f1a-a78e-444b-ab89-118249089726\" (UID: \"43b70f1a-a78e-444b-ab89-118249089726\") "
Sep 16 19:47:15.312198 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.312032    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b70f1a-a78e-444b-ab89-118249089726-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43b70f1a-a78e-444b-ab89-118249089726" (UID: "43b70f1a-a78e-444b-ab89-118249089726"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:47:15.312304 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.312272    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-model-cache" (OuterVolumeSpecName: "model-cache") pod "43b70f1a-a78e-444b-ab89-118249089726" (UID: "43b70f1a-a78e-444b-ab89-118249089726"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:15.312506 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.312443    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-home" (OuterVolumeSpecName: "home") pod "43b70f1a-a78e-444b-ab89-118249089726" (UID: "43b70f1a-a78e-444b-ab89-118249089726"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:15.314281 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.314253    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b70f1a-a78e-444b-ab89-118249089726-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43b70f1a-a78e-444b-ab89-118249089726" (UID: "43b70f1a-a78e-444b-ab89-118249089726"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:47:15.314499 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.314467    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b70f1a-a78e-444b-ab89-118249089726-kube-api-access-zzc2n" (OuterVolumeSpecName: "kube-api-access-zzc2n") pod "43b70f1a-a78e-444b-ab89-118249089726" (UID: "43b70f1a-a78e-444b-ab89-118249089726"). InnerVolumeSpecName "kube-api-access-zzc2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:15.314610 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.314525    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-dshm" (OuterVolumeSpecName: "dshm") pod "43b70f1a-a78e-444b-ab89-118249089726" (UID: "43b70f1a-a78e-444b-ab89-118249089726"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:15.326987 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.326953    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43b70f1a-a78e-444b-ab89-118249089726" (UID: "43b70f1a-a78e-444b-ab89-118249089726"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:15.337667 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.337636    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43b70f1a-a78e-444b-ab89-118249089726" (UID: "43b70f1a-a78e-444b-ab89-118249089726"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:15.413054 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.413013    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:15.413054 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.413045    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43b70f1a-a78e-444b-ab89-118249089726-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:15.413054 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.413056    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzc2n\" (UniqueName: \"kubernetes.io/projected/43b70f1a-a78e-444b-ab89-118249089726-kube-api-access-zzc2n\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:15.413054 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.413064    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:15.413343 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.413073    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70f1a-a78e-444b-ab89-118249089726-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:15.413343 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.413080    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:15.413343 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.413087    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:15.413343 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.413094    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70f1a-a78e-444b-ab89-118249089726-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:15.629730 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.629638    2590 generic.go:356] "Generic (PLEG): container finished" podID="43b70f1a-a78e-444b-ab89-118249089726" containerID="eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2" exitCode=0
Sep 16 19:47:15.629730 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.629702    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"
Sep 16 19:47:15.630290 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.629721    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" event={"ID":"43b70f1a-a78e-444b-ab89-118249089726","Type":"ContainerDied","Data":"eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2"}
Sep 16 19:47:15.630290 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.629812    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x" event={"ID":"43b70f1a-a78e-444b-ab89-118249089726","Type":"ContainerDied","Data":"d7be42c61a15034d8d6f6f1fa45d77bd8dc7fd9cc6f111d5f37b237eee555b24"}
Sep 16 19:47:15.630290 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.629837    2590 scope.go:117] "RemoveContainer" containerID="eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2"
Sep 16 19:47:15.638847 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.638825    2590 scope.go:117] "RemoveContainer" containerID="f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7"
Sep 16 19:47:15.646964 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.646945    2590 scope.go:117] "RemoveContainer" containerID="eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2"
Sep 16 19:47:15.647297 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:47:15.647277    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2\": container with ID starting with eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2 not found: ID does not exist" containerID="eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2"
Sep 16 19:47:15.647383 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.647311    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2"} err="failed to get container status \"eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2\": rpc error: code = NotFound desc = could not find container \"eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2\": container with ID starting with eeb4332ad64f3aece978c7ea3aaa3b402d177f94f4db743504d253ab55f5dbf2 not found: ID does not exist"
Sep 16 19:47:15.647383 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.647337    2590 scope.go:117] "RemoveContainer" containerID="f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7"
Sep 16 19:47:15.647640 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:47:15.647621    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7\": container with ID starting with f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7 not found: ID does not exist" containerID="f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7"
Sep 16 19:47:15.647678 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.647648    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7"} err="failed to get container status \"f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7\": rpc error: code = NotFound desc = could not find container \"f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7\": container with ID starting with f3acffe38ad7e262ae162c13348e98dd948c03573e3a768743949b956eb01bf7 not found: ID does not exist"
Sep 16 19:47:15.658138 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.658101    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"]
Sep 16 19:47:15.661799 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:15.661766    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-64b9dd657b-kdz7x"]
Sep 16 19:47:16.852941 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:16.852906    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" podUID="aaabae5a-c1e7-4f41-a483-087846bd2825" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:47:17.025903 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:17.025858    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43b70f1a-a78e-444b-ab89-118249089726" path="/var/lib/kubelet/pods/43b70f1a-a78e-444b-ab89-118249089726/volumes"
Sep 16 19:47:18.710585 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:18.710549    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:18.806360 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:18.806314    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:26.852892 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:26.852851    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" podUID="aaabae5a-c1e7-4f41-a483-087846bd2825" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:47:26.853285 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:26.852981    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:47:35.861751 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:35.861710    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:35.936822 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:35.936788    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:36.853551 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:36.853513    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" podUID="aaabae5a-c1e7-4f41-a483-087846bd2825" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:47:44.702953 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.702925    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:47:44.753377 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.753293    2590 generic.go:356] "Generic (PLEG): container finished" podID="aaabae5a-c1e7-4f41-a483-087846bd2825" containerID="b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c" exitCode=0
Sep 16 19:47:44.753377 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.753335    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"
Sep 16 19:47:44.753556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.753372    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" event={"ID":"aaabae5a-c1e7-4f41-a483-087846bd2825","Type":"ContainerDied","Data":"b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c"}
Sep 16 19:47:44.753556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.753424    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq" event={"ID":"aaabae5a-c1e7-4f41-a483-087846bd2825","Type":"ContainerDied","Data":"e2fa462073360df8929e1ab5b165d018bfe8c119105ce22308b7c9bd9691eba8"}
Sep 16 19:47:44.753556 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.753448    2590 scope.go:117] "RemoveContainer" containerID="b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c"
Sep 16 19:47:44.761610 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.761594    2590 scope.go:117] "RemoveContainer" containerID="b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c"
Sep 16 19:47:44.761934 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:47:44.761915    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c\": container with ID starting with b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c not found: ID does not exist" containerID="b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c"
Sep 16 19:47:44.762000 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.761945    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c"} err="failed to get container status \"b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c\": rpc error: code = NotFound desc = could not find container \"b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c\": container with ID starting with b4a5a942def7af53ed02e7817f8c93e201f29ed1416ae57180216c87bc161c6c not found: ID does not exist"
Sep 16 19:47:44.763100 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.763083    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8pn8\" (UniqueName: \"kubernetes.io/projected/aaabae5a-c1e7-4f41-a483-087846bd2825-kube-api-access-h8pn8\") pod \"aaabae5a-c1e7-4f41-a483-087846bd2825\" (UID: \"aaabae5a-c1e7-4f41-a483-087846bd2825\") "
Sep 16 19:47:44.763145 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.763125    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaabae5a-c1e7-4f41-a483-087846bd2825-tls-certs\") pod \"aaabae5a-c1e7-4f41-a483-087846bd2825\" (UID: \"aaabae5a-c1e7-4f41-a483-087846bd2825\") "
Sep 16 19:47:44.765262 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.765238    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaabae5a-c1e7-4f41-a483-087846bd2825-kube-api-access-h8pn8" (OuterVolumeSpecName: "kube-api-access-h8pn8") pod "aaabae5a-c1e7-4f41-a483-087846bd2825" (UID: "aaabae5a-c1e7-4f41-a483-087846bd2825"). InnerVolumeSpecName "kube-api-access-h8pn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:44.765371 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.765310    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaabae5a-c1e7-4f41-a483-087846bd2825-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aaabae5a-c1e7-4f41-a483-087846bd2825" (UID: "aaabae5a-c1e7-4f41-a483-087846bd2825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:47:44.863981 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.863943    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8pn8\" (UniqueName: \"kubernetes.io/projected/aaabae5a-c1e7-4f41-a483-087846bd2825-kube-api-access-h8pn8\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:44.863981 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:44.863974    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaabae5a-c1e7-4f41-a483-087846bd2825-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:47:45.071037 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:45.071001    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"]
Sep 16 19:47:45.082024 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:45.081994    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-92qbq"]
Sep 16 19:47:47.025482 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:47.025448    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaabae5a-c1e7-4f41-a483-087846bd2825" path="/var/lib/kubelet/pods/aaabae5a-c1e7-4f41-a483-087846bd2825/volumes"
Sep 16 19:47:49.684667 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:49.684637    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:49.780643 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:49.780609    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:52.460581 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460539    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"]
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460811    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460822    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460833    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460839    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460847    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaabae5a-c1e7-4f41-a483-087846bd2825" containerName="main"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460853    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaabae5a-c1e7-4f41-a483-087846bd2825" containerName="main"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460863    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460868    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460873    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="storage-initializer"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460893    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="storage-initializer"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460903    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="storage-initializer"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460908    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="storage-initializer"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460914    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460919    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460972    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460979    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaabae5a-c1e7-4f41-a483-087846bd2825" containerName="main"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460985    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b70f1a-a78e-444b-ab89-118249089726" containerName="main"
Sep 16 19:47:52.461007 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.460991    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="154b049b-2058-450f-b9f5-598d024e071f" containerName="main"
Sep 16 19:47:52.465334 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.465315    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.469427 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.469401    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 19:47:52.469585 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.469410    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-lb66q\""
Sep 16 19:47:52.469585 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.469441    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 19:47:52.469585 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.469410    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 19:47:52.473934 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.473914    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"]
Sep 16 19:47:52.522133 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.522091    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.522133 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.522144    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876sw\" (UniqueName: \"kubernetes.io/projected/e9943641-9bfe-4f6e-8e0e-376369d25fa1-kube-api-access-876sw\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.522379 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.522199    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.522379 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.522241    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.522379 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.522265    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.522379 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.522299    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.622971 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.622927    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.623171 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.622977    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.623171 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.623016    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.623171 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.623060    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.623331 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.623194    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-876sw\" (UniqueName: \"kubernetes.io/projected/e9943641-9bfe-4f6e-8e0e-376369d25fa1-kube-api-access-876sw\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.623331 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.623252    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.623432 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.623378    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.623432 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.623419    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.623530 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.623448    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.625609 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.625578    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.625731 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.625714    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.633413 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.633391    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-876sw\" (UniqueName: \"kubernetes.io/projected/e9943641-9bfe-4f6e-8e0e-376369d25fa1-kube-api-access-876sw\") pod \"rolling-upgrade-test-kserve-74b79d7f77-k5ctp\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:52.775826 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:52.775785    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:53.066500 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:53.066473    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"]
Sep 16 19:47:53.075392 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:47:53.075356    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9943641_9bfe_4f6e_8e0e_376369d25fa1.slice/crio-5c5cc0ca8957811ee39409d69c3c6193ad63ebaf7c5bb22fa6e52afb406d90d4 WatchSource:0}: Error finding container 5c5cc0ca8957811ee39409d69c3c6193ad63ebaf7c5bb22fa6e52afb406d90d4: Status 404 returned error can't find the container with id 5c5cc0ca8957811ee39409d69c3c6193ad63ebaf7c5bb22fa6e52afb406d90d4
Sep 16 19:47:53.791537 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:53.791493    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp" event={"ID":"e9943641-9bfe-4f6e-8e0e-376369d25fa1","Type":"ContainerStarted","Data":"6ef08bef7777ad065b909a586a244accbb9a121ebd2d72c316a0447af9b7ddc7"}
Sep 16 19:47:53.792024 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:53.791599    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp" event={"ID":"e9943641-9bfe-4f6e-8e0e-376369d25fa1","Type":"ContainerStarted","Data":"5c5cc0ca8957811ee39409d69c3c6193ad63ebaf7c5bb22fa6e52afb406d90d4"}
Sep 16 19:47:53.812569 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:53.812515    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp" podStartSLOduration=1.812498466 podStartE2EDuration="1.812498466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:53.810748637 +0000 UTC m=+4089.335207471" watchObservedRunningTime="2026-09-16 19:47:53.812498466 +0000 UTC m=+4089.336957299"
Sep 16 19:47:54.776322 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:54.776277    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:47:56.041706 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:47:56.041672    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:02.397276 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:02.397240    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:02.470652 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:02.470595    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:02.776094 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:02.776057    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:48:02.788674 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:02.788641    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:48:02.841787 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:02.841746    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:48:05.489157 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.489124    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:05.760440 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.760359    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"]
Sep 16 19:48:05.760914 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.760868    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e82897-eb31-487e-97f5-147b9fdc1c35" containerName="istio-proxy"
Sep 16 19:48:05.764099 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.764080    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:05.766805 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.766787    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-fc48g\""
Sep 16 19:48:05.777112 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.777087    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"]
Sep 16 19:48:05.792150 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.792121    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:05.833173 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.833136    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsc5s\" (UniqueName: \"kubernetes.io/projected/4a86fa98-8d28-4873-8a74-0c40a02379b3-kube-api-access-wsc5s\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv\" (UID: \"4a86fa98-8d28-4873-8a74-0c40a02379b3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:05.833350 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.833207    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86fa98-8d28-4873-8a74-0c40a02379b3-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv\" (UID: \"4a86fa98-8d28-4873-8a74-0c40a02379b3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:05.933651 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.933604    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsc5s\" (UniqueName: \"kubernetes.io/projected/4a86fa98-8d28-4873-8a74-0c40a02379b3-kube-api-access-wsc5s\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv\" (UID: \"4a86fa98-8d28-4873-8a74-0c40a02379b3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:05.933841 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.933672    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86fa98-8d28-4873-8a74-0c40a02379b3-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv\" (UID: \"4a86fa98-8d28-4873-8a74-0c40a02379b3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:05.936354 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.936334    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86fa98-8d28-4873-8a74-0c40a02379b3-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv\" (UID: \"4a86fa98-8d28-4873-8a74-0c40a02379b3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:05.943766 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:05.943737    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsc5s\" (UniqueName: \"kubernetes.io/projected/4a86fa98-8d28-4873-8a74-0c40a02379b3-kube-api-access-wsc5s\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv\" (UID: \"4a86fa98-8d28-4873-8a74-0c40a02379b3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:06.074486 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:06.074452    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:06.350168 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:06.350133    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"]
Sep 16 19:48:06.352451 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:48:06.352409    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a86fa98_8d28_4873_8a74_0c40a02379b3.slice/crio-34c1c7b86d455ca667eaa71a1c12be978ab99a8430baa69ef339a97d36459157 WatchSource:0}: Error finding container 34c1c7b86d455ca667eaa71a1c12be978ab99a8430baa69ef339a97d36459157: Status 404 returned error can't find the container with id 34c1c7b86d455ca667eaa71a1c12be978ab99a8430baa69ef339a97d36459157
Sep 16 19:48:06.855626 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:06.855587    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" event={"ID":"4a86fa98-8d28-4873-8a74-0c40a02379b3","Type":"ContainerStarted","Data":"4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6"}
Sep 16 19:48:06.856122 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:06.855649    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" event={"ID":"4a86fa98-8d28-4873-8a74-0c40a02379b3","Type":"ContainerStarted","Data":"34c1c7b86d455ca667eaa71a1c12be978ab99a8430baa69ef339a97d36459157"}
Sep 16 19:48:06.856122 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:06.855681    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:06.878219 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:06.878167    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" podStartSLOduration=1.878151158 podStartE2EDuration="1.878151158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:06.873768357 +0000 UTC m=+4102.398227190" watchObservedRunningTime="2026-09-16 19:48:06.878151158 +0000 UTC m=+4102.402609991"
Sep 16 19:48:19.781525 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:19.781482    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:37.862848 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:37.862762    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:48:39.789262 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.789217    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"]
Sep 16 19:48:39.792632 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.792606    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.795335 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.795316    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 19:48:39.795555 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.795537    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 19:48:39.795623 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.795587    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 19:48:39.796805 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.796769    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 19:48:39.796946 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.796834    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-dh48s\""
Sep 16 19:48:39.796946 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.796866    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:48:39.806736 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.806712    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"]
Sep 16 19:48:39.831386 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.831356    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tmp-dir\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.831555 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.831400    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-dshm\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.831555 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.831423    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52a2e71b-86f8-4425-8a2b-9038e2409ce8-cabundle-cert\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.831555 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.831495    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-home\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.831555 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.831520    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.831555 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.831549    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-model-cache\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.831767 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.831574    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tls-certs\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.831767 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.831614    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvf5r\" (UniqueName: \"kubernetes.io/projected/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kube-api-access-xvf5r\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932043    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-home\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932084 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932087    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932115    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-model-cache\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932149    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tls-certs\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932191    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvf5r\" (UniqueName: \"kubernetes.io/projected/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kube-api-access-xvf5r\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932226    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tmp-dir\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932256    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-dshm\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932369 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932276    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52a2e71b-86f8-4425-8a2b-9038e2409ce8-cabundle-cert\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932549    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-home\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932598    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932642    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-model-cache\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.932851 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.932716    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tmp-dir\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.933061 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.933003    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52a2e71b-86f8-4425-8a2b-9038e2409ce8-cabundle-cert\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.935011 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.934989    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tls-certs\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.935261 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.935241    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-dshm\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:39.942071 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:39.942045    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvf5r\" (UniqueName: \"kubernetes.io/projected/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kube-api-access-xvf5r\") pod \"prestop-hook-test-kserve-6f76f59c4d-cgj65\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:40.103854 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:40.103762    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:40.384056 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:40.384026    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"]
Sep 16 19:48:40.385430 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:48:40.385407    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a2e71b_86f8_4425_8a2b_9038e2409ce8.slice/crio-9db5b9e9359c3a68bdf348d2496eb7d3e650a89981d326f45f93a1faad209ea5 WatchSource:0}: Error finding container 9db5b9e9359c3a68bdf348d2496eb7d3e650a89981d326f45f93a1faad209ea5: Status 404 returned error can't find the container with id 9db5b9e9359c3a68bdf348d2496eb7d3e650a89981d326f45f93a1faad209ea5
Sep 16 19:48:41.001008 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:41.000961    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" event={"ID":"52a2e71b-86f8-4425-8a2b-9038e2409ce8","Type":"ContainerStarted","Data":"9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946"}
Sep 16 19:48:41.001452 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:41.001032    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" event={"ID":"52a2e71b-86f8-4425-8a2b-9038e2409ce8","Type":"ContainerStarted","Data":"9db5b9e9359c3a68bdf348d2496eb7d3e650a89981d326f45f93a1faad209ea5"}
Sep 16 19:48:43.011532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:43.011438    2590 generic.go:356] "Generic (PLEG): container finished" podID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerID="9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946" exitCode=0
Sep 16 19:48:43.011532 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:43.011516    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" event={"ID":"52a2e71b-86f8-4425-8a2b-9038e2409ce8","Type":"ContainerDied","Data":"9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946"}
Sep 16 19:48:44.019328 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:44.019289    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" event={"ID":"52a2e71b-86f8-4425-8a2b-9038e2409ce8","Type":"ContainerStarted","Data":"4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f"}
Sep 16 19:48:44.051521 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:44.051460    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" podStartSLOduration=2.868887022 podStartE2EDuration="5.051442161s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.182555139s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:44.049292041 +0000 UTC m=+4139.573750875" watchObservedRunningTime="2026-09-16 19:48:44.051442161 +0000 UTC m=+4139.575900994"
Sep 16 19:48:44.104242 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:44.104204    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:50.104434 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:50.104380    2590 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:48:50.105734 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:48:50.105705    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" podUID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:49:00.104840 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:00.104785    2590 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" podUID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:49:06.073658 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:06.073627    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"]
Sep 16 19:49:06.074296 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:06.074055    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:06.155286 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:06.155243    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:10.774841 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:10.774798    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:10.793744 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:10.793700    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"]
Sep 16 19:49:10.810656 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:10.810624    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"]
Sep 16 19:49:10.810957 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:49:10.810915    2590 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:49:10.810957 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:49:10.810957 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:49:10.810957 ip-10-0-143-44 kubenswrapper[2590]:  > containerID="6ef08bef7777ad065b909a586a244accbb9a121ebd2d72c316a0447af9b7ddc7" cmd=["/bin/sleep","15"]
Sep 16 19:49:10.811241 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:10.810987    2590 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp" message="" err=<
Sep 16 19:49:10.811241 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:49:10.811241 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:49:10.811241 ip-10-0-143-44 kubenswrapper[2590]:  >
Sep 16 19:49:10.811241 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:49:10.811006    2590 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:49:10.811241 ip-10-0-143-44 kubenswrapper[2590]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:49:10.811241 ip-10-0-143-44 kubenswrapper[2590]:         , stdout: , stderr: , exit code -1
Sep 16 19:49:10.811241 ip-10-0-143-44 kubenswrapper[2590]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp" podUID="e9943641-9bfe-4f6e-8e0e-376369d25fa1" containerName="main" containerID="cri-o://6ef08bef7777ad065b909a586a244accbb9a121ebd2d72c316a0447af9b7ddc7"
Sep 16 19:49:10.811241 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:10.811059    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp" podUID="e9943641-9bfe-4f6e-8e0e-376369d25fa1" containerName="main" containerID="cri-o://6ef08bef7777ad065b909a586a244accbb9a121ebd2d72c316a0447af9b7ddc7" gracePeriod=60
Sep 16 19:49:10.831541 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:10.831505    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp" podUID="e9943641-9bfe-4f6e-8e0e-376369d25fa1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:49:10.865583 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:10.865546    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:11.145571 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.145541    2590 generic.go:356] "Generic (PLEG): container finished" podID="e9943641-9bfe-4f6e-8e0e-376369d25fa1" containerID="6ef08bef7777ad065b909a586a244accbb9a121ebd2d72c316a0447af9b7ddc7" exitCode=0
Sep 16 19:49:11.145717 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.145634    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp" event={"ID":"e9943641-9bfe-4f6e-8e0e-376369d25fa1","Type":"ContainerDied","Data":"6ef08bef7777ad065b909a586a244accbb9a121ebd2d72c316a0447af9b7ddc7"}
Sep 16 19:49:11.145717 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.145700    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp" event={"ID":"e9943641-9bfe-4f6e-8e0e-376369d25fa1","Type":"ContainerDied","Data":"5c5cc0ca8957811ee39409d69c3c6193ad63ebaf7c5bb22fa6e52afb406d90d4"}
Sep 16 19:49:11.145813 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.145726    2590 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5cc0ca8957811ee39409d69c3c6193ad63ebaf7c5bb22fa6e52afb406d90d4"
Sep 16 19:49:11.151569 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.151543    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:49:11.303366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303329    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tmp-dir\") pod \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") "
Sep 16 19:49:11.303366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303368    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tls-certs\") pod \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") "
Sep 16 19:49:11.303607 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303390    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-home\") pod \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") "
Sep 16 19:49:11.303607 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303435    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-dshm\") pod \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") "
Sep 16 19:49:11.303607 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303552    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-model-cache\") pod \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") "
Sep 16 19:49:11.303771 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303633    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9943641-9bfe-4f6e-8e0e-376369d25fa1" (UID: "e9943641-9bfe-4f6e-8e0e-376369d25fa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:11.303771 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303641    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-home" (OuterVolumeSpecName: "home") pod "e9943641-9bfe-4f6e-8e0e-376369d25fa1" (UID: "e9943641-9bfe-4f6e-8e0e-376369d25fa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:11.303771 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303676    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-876sw\" (UniqueName: \"kubernetes.io/projected/e9943641-9bfe-4f6e-8e0e-376369d25fa1-kube-api-access-876sw\") pod \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\" (UID: \"e9943641-9bfe-4f6e-8e0e-376369d25fa1\") "
Sep 16 19:49:11.303951 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303839    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9943641-9bfe-4f6e-8e0e-376369d25fa1" (UID: "e9943641-9bfe-4f6e-8e0e-376369d25fa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:11.304013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.303980    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.304013 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.304001    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.304183 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.304016    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.306059 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.305978    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9943641-9bfe-4f6e-8e0e-376369d25fa1-kube-api-access-876sw" (OuterVolumeSpecName: "kube-api-access-876sw") pod "e9943641-9bfe-4f6e-8e0e-376369d25fa1" (UID: "e9943641-9bfe-4f6e-8e0e-376369d25fa1"). InnerVolumeSpecName "kube-api-access-876sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:11.306059 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.305983    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9943641-9bfe-4f6e-8e0e-376369d25fa1" (UID: "e9943641-9bfe-4f6e-8e0e-376369d25fa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:11.306059 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.306031    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-dshm" (OuterVolumeSpecName: "dshm") pod "e9943641-9bfe-4f6e-8e0e-376369d25fa1" (UID: "e9943641-9bfe-4f6e-8e0e-376369d25fa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:11.404679 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.404587    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-876sw\" (UniqueName: \"kubernetes.io/projected/e9943641-9bfe-4f6e-8e0e-376369d25fa1-kube-api-access-876sw\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.404679 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.404626    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9943641-9bfe-4f6e-8e0e-376369d25fa1-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.404679 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:11.404641    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9943641-9bfe-4f6e-8e0e-376369d25fa1-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:12.150066 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:12.150033    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"
Sep 16 19:49:12.171839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:12.171801    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"]
Sep 16 19:49:12.178516 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:12.178484    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-k5ctp"]
Sep 16 19:49:13.025970 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:13.025933    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9943641-9bfe-4f6e-8e0e-376369d25fa1" path="/var/lib/kubelet/pods/e9943641-9bfe-4f6e-8e0e-376369d25fa1/volumes"
Sep 16 19:49:17.863176 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:17.863129    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" podUID="4a86fa98-8d28-4873-8a74-0c40a02379b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:21.096805 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:21.096744    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" podUID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerName="main" containerID="cri-o://4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f" gracePeriod=45
Sep 16 19:49:23.039754 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.039731    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6f76f59c4d-cgj65_52a2e71b-86f8-4425-8a2b-9038e2409ce8/main/0.log"
Sep 16 19:49:23.042480 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.042460    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:49:23.199450 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.199367    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6f76f59c4d-cgj65_52a2e71b-86f8-4425-8a2b-9038e2409ce8/main/0.log"
Sep 16 19:49:23.199811 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.199781    2590 generic.go:356] "Generic (PLEG): container finished" podID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerID="4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f" exitCode=130
Sep 16 19:49:23.199944 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.199833    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"
Sep 16 19:49:23.199944 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.199871    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" event={"ID":"52a2e71b-86f8-4425-8a2b-9038e2409ce8","Type":"ContainerDied","Data":"4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f"}
Sep 16 19:49:23.200034 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.199949    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65" event={"ID":"52a2e71b-86f8-4425-8a2b-9038e2409ce8","Type":"ContainerDied","Data":"9db5b9e9359c3a68bdf348d2496eb7d3e650a89981d326f45f93a1faad209ea5"}
Sep 16 19:49:23.200034 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.199974    2590 scope.go:117] "RemoveContainer" containerID="4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f"
Sep 16 19:49:23.206462 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206440    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-home\") pod \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") "
Sep 16 19:49:23.206569 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206480    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-model-cache\") pod \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") "
Sep 16 19:49:23.206569 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206517    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tls-certs\") pod \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") "
Sep 16 19:49:23.206569 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206556    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52a2e71b-86f8-4425-8a2b-9038e2409ce8-cabundle-cert\") pod \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") "
Sep 16 19:49:23.206706 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206583    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kserve-provision-location\") pod \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") "
Sep 16 19:49:23.206767 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206734    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvf5r\" (UniqueName: \"kubernetes.io/projected/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kube-api-access-xvf5r\") pod \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") "
Sep 16 19:49:23.206820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206736    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-model-cache" (OuterVolumeSpecName: "model-cache") pod "52a2e71b-86f8-4425-8a2b-9038e2409ce8" (UID: "52a2e71b-86f8-4425-8a2b-9038e2409ce8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:23.206820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206776    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-home" (OuterVolumeSpecName: "home") pod "52a2e71b-86f8-4425-8a2b-9038e2409ce8" (UID: "52a2e71b-86f8-4425-8a2b-9038e2409ce8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:23.206820 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206788    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-dshm\") pod \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") "
Sep 16 19:49:23.207020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206902    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tmp-dir\") pod \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\" (UID: \"52a2e71b-86f8-4425-8a2b-9038e2409ce8\") "
Sep 16 19:49:23.207020 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.206973    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a2e71b-86f8-4425-8a2b-9038e2409ce8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52a2e71b-86f8-4425-8a2b-9038e2409ce8" (UID: "52a2e71b-86f8-4425-8a2b-9038e2409ce8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:49:23.207187 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.207165    2590 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-home\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:23.207187 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.207190    2590 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-model-cache\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:23.207187 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.207204    2590 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52a2e71b-86f8-4425-8a2b-9038e2409ce8-cabundle-cert\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:23.207486 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.207424    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52a2e71b-86f8-4425-8a2b-9038e2409ce8" (UID: "52a2e71b-86f8-4425-8a2b-9038e2409ce8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:23.209071 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.209041    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52a2e71b-86f8-4425-8a2b-9038e2409ce8" (UID: "52a2e71b-86f8-4425-8a2b-9038e2409ce8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:23.209303 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.209273    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kube-api-access-xvf5r" (OuterVolumeSpecName: "kube-api-access-xvf5r") pod "52a2e71b-86f8-4425-8a2b-9038e2409ce8" (UID: "52a2e71b-86f8-4425-8a2b-9038e2409ce8"). InnerVolumeSpecName "kube-api-access-xvf5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:23.209389 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.209292    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-dshm" (OuterVolumeSpecName: "dshm") pod "52a2e71b-86f8-4425-8a2b-9038e2409ce8" (UID: "52a2e71b-86f8-4425-8a2b-9038e2409ce8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:23.209972 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.209949    2590 scope.go:117] "RemoveContainer" containerID="9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946"
Sep 16 19:49:23.229450 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.229431    2590 scope.go:117] "RemoveContainer" containerID="4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f"
Sep 16 19:49:23.229821 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:49:23.229801    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f\": container with ID starting with 4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f not found: ID does not exist" containerID="4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f"
Sep 16 19:49:23.229905 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.229830    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f"} err="failed to get container status \"4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f\": rpc error: code = NotFound desc = could not find container \"4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f\": container with ID starting with 4e80de3094158fe660f29c7c2b1cac4b4ec786d68f75cd1c0ff01b34771d7a8f not found: ID does not exist"
Sep 16 19:49:23.229905 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.229850    2590 scope.go:117] "RemoveContainer" containerID="9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946"
Sep 16 19:49:23.230166 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:49:23.230151    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946\": container with ID starting with 9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946 not found: ID does not exist" containerID="9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946"
Sep 16 19:49:23.230212 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.230171    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946"} err="failed to get container status \"9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946\": rpc error: code = NotFound desc = could not find container \"9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946\": container with ID starting with 9d6fb574bce24db8f6842de73db33a55e9e562569e69d7c0ae5e0b1c71f06946 not found: ID does not exist"
Sep 16 19:49:23.237477 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.237450    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52a2e71b-86f8-4425-8a2b-9038e2409ce8" (UID: "52a2e71b-86f8-4425-8a2b-9038e2409ce8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:23.307738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.307700    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:23.307738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.307731    2590 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kserve-provision-location\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:23.307738 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.307742    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvf5r\" (UniqueName: \"kubernetes.io/projected/52a2e71b-86f8-4425-8a2b-9038e2409ce8-kube-api-access-xvf5r\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:23.308021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.307750    2590 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-dshm\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:23.308021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.307758    2590 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a2e71b-86f8-4425-8a2b-9038e2409ce8-tmp-dir\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:23.530578 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.530543    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"]
Sep 16 19:49:23.533526 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:23.533499    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-cgj65"]
Sep 16 19:49:25.025337 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:25.025305    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" path="/var/lib/kubelet/pods/52a2e71b-86f8-4425-8a2b-9038e2409ce8/volumes"
Sep 16 19:49:25.830057 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:25.830017    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" podUID="4a86fa98-8d28-4873-8a74-0c40a02379b3" containerName="main" containerID="cri-o://4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6" gracePeriod=45
Sep 16 19:49:27.862142 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:27.862107    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" podUID="4a86fa98-8d28-4873-8a74-0c40a02379b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:37.862577 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:37.862539    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" podUID="4a86fa98-8d28-4873-8a74-0c40a02379b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:37.863064 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:37.862641    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:49:47.863096 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:47.863054    2590 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" podUID="4a86fa98-8d28-4873-8a74-0c40a02379b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:53.467071 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:53.467035    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:56.081467 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.081441    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:49:56.167951 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.167859    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86fa98-8d28-4873-8a74-0c40a02379b3-tls-certs\") pod \"4a86fa98-8d28-4873-8a74-0c40a02379b3\" (UID: \"4a86fa98-8d28-4873-8a74-0c40a02379b3\") "
Sep 16 19:49:56.168124 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.167959    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsc5s\" (UniqueName: \"kubernetes.io/projected/4a86fa98-8d28-4873-8a74-0c40a02379b3-kube-api-access-wsc5s\") pod \"4a86fa98-8d28-4873-8a74-0c40a02379b3\" (UID: \"4a86fa98-8d28-4873-8a74-0c40a02379b3\") "
Sep 16 19:49:56.170201 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.170174    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a86fa98-8d28-4873-8a74-0c40a02379b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a86fa98-8d28-4873-8a74-0c40a02379b3" (UID: "4a86fa98-8d28-4873-8a74-0c40a02379b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:56.170307 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.170195    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a86fa98-8d28-4873-8a74-0c40a02379b3-kube-api-access-wsc5s" (OuterVolumeSpecName: "kube-api-access-wsc5s") pod "4a86fa98-8d28-4873-8a74-0c40a02379b3" (UID: "4a86fa98-8d28-4873-8a74-0c40a02379b3"). InnerVolumeSpecName "kube-api-access-wsc5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:56.268838 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.268797    2590 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86fa98-8d28-4873-8a74-0c40a02379b3-tls-certs\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:56.268838 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.268834    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsc5s\" (UniqueName: \"kubernetes.io/projected/4a86fa98-8d28-4873-8a74-0c40a02379b3-kube-api-access-wsc5s\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:49:56.346416 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.346383    2590 generic.go:356] "Generic (PLEG): container finished" podID="4a86fa98-8d28-4873-8a74-0c40a02379b3" containerID="4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6" exitCode=0
Sep 16 19:49:56.346592 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.346423    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"
Sep 16 19:49:56.346592 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.346445    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" event={"ID":"4a86fa98-8d28-4873-8a74-0c40a02379b3","Type":"ContainerDied","Data":"4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6"}
Sep 16 19:49:56.346592 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.346491    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv" event={"ID":"4a86fa98-8d28-4873-8a74-0c40a02379b3","Type":"ContainerDied","Data":"34c1c7b86d455ca667eaa71a1c12be978ab99a8430baa69ef339a97d36459157"}
Sep 16 19:49:56.346592 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.346512    2590 scope.go:117] "RemoveContainer" containerID="4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6"
Sep 16 19:49:56.355725 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.355706    2590 scope.go:117] "RemoveContainer" containerID="4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6"
Sep 16 19:49:56.356027 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:49:56.356008    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6\": container with ID starting with 4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6 not found: ID does not exist" containerID="4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6"
Sep 16 19:49:56.356095 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.356035    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6"} err="failed to get container status \"4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6\": rpc error: code = NotFound desc = could not find container \"4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6\": container with ID starting with 4b6c255d435ddb73759c791217a4fd41c9d50acfa9b9786ab9e12304cc19abb6 not found: ID does not exist"
Sep 16 19:49:56.376942 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.376905    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"]
Sep 16 19:49:56.386315 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:56.386290    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r2npv"]
Sep 16 19:49:57.026307 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:57.026271    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a86fa98-8d28-4873-8a74-0c40a02379b3" path="/var/lib/kubelet/pods/4a86fa98-8d28-4873-8a74-0c40a02379b3/volumes"
Sep 16 19:49:57.188608 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:49:57.188572    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:00.277952 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.277909    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:00.653068 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.652975    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:00.666558 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666522    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-svfwh/must-gather-wslk4"]
Sep 16 19:50:00.666823 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666811    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9943641-9bfe-4f6e-8e0e-376369d25fa1" containerName="main"
Sep 16 19:50:00.666867 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666825    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9943641-9bfe-4f6e-8e0e-376369d25fa1" containerName="main"
Sep 16 19:50:00.666867 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666843    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerName="storage-initializer"
Sep 16 19:50:00.666867 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666849    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerName="storage-initializer"
Sep 16 19:50:00.666867 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666859    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a86fa98-8d28-4873-8a74-0c40a02379b3" containerName="main"
Sep 16 19:50:00.667021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666869    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a86fa98-8d28-4873-8a74-0c40a02379b3" containerName="main"
Sep 16 19:50:00.667021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666893    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerName="main"
Sep 16 19:50:00.667021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666899    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerName="main"
Sep 16 19:50:00.667021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666967    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a86fa98-8d28-4873-8a74-0c40a02379b3" containerName="main"
Sep 16 19:50:00.667021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666977    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9943641-9bfe-4f6e-8e0e-376369d25fa1" containerName="main"
Sep 16 19:50:00.667021 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.666983    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a2e71b-86f8-4425-8a2b-9038e2409ce8" containerName="main"
Sep 16 19:50:00.671390 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.671371    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:00.679253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.679232    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-svfwh\"/\"default-dockercfg-nf297\""
Sep 16 19:50:00.679827 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.679809    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-svfwh\"/\"openshift-service-ca.crt\""
Sep 16 19:50:00.681717 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.681704    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-svfwh\"/\"kube-root-ca.crt\""
Sep 16 19:50:00.700240 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.700206    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-svfwh/must-gather-wslk4"]
Sep 16 19:50:00.771342 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.771302    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:00.802700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.802660    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wv9t\" (UniqueName: \"kubernetes.io/projected/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-kube-api-access-4wv9t\") pod \"must-gather-wslk4\" (UID: \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\") " pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:00.802700 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.802706    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-must-gather-output\") pod \"must-gather-wslk4\" (UID: \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\") " pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:00.903495 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.903382    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wv9t\" (UniqueName: \"kubernetes.io/projected/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-kube-api-access-4wv9t\") pod \"must-gather-wslk4\" (UID: \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\") " pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:00.903495 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.903430    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-must-gather-output\") pod \"must-gather-wslk4\" (UID: \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\") " pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:00.903739 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.903710    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-must-gather-output\") pod \"must-gather-wslk4\" (UID: \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\") " pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:00.913209 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.913178    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wv9t\" (UniqueName: \"kubernetes.io/projected/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-kube-api-access-4wv9t\") pod \"must-gather-wslk4\" (UID: \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\") " pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:00.980126 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:00.980091    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:01.251749 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:01.251720    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-svfwh/must-gather-wslk4"]
Sep 16 19:50:01.254393 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:50:01.254364    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c16b375_ece4_4a5d_9cbb_78eecad4c7e2.slice/crio-490569f6430cfe584722b97f1dc3a1b89ae014b62adae756801e03d0fba91931 WatchSource:0}: Error finding container 490569f6430cfe584722b97f1dc3a1b89ae014b62adae756801e03d0fba91931: Status 404 returned error can't find the container with id 490569f6430cfe584722b97f1dc3a1b89ae014b62adae756801e03d0fba91931
Sep 16 19:50:01.256325 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:01.256303    2590 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:50:01.372559 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:01.372518    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svfwh/must-gather-wslk4" event={"ID":"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2","Type":"ContainerStarted","Data":"490569f6430cfe584722b97f1dc3a1b89ae014b62adae756801e03d0fba91931"}
Sep 16 19:50:02.039160 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:02.039067    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:04.122701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:04.122664    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:06.936650 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:06.936560    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:07.036412 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:07.036362    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:08.412785 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:08.412752    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svfwh/must-gather-wslk4" event={"ID":"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2","Type":"ContainerStarted","Data":"de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108"}
Sep 16 19:50:08.413199 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:08.412794    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svfwh/must-gather-wslk4" event={"ID":"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2","Type":"ContainerStarted","Data":"7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8"}
Sep 16 19:50:08.451739 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:08.451684    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-svfwh/must-gather-wslk4" podStartSLOduration=2.160064425 podStartE2EDuration="8.451668035s" totalImagesPullingTime="6.29160361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:08.450396954 +0000 UTC m=+4223.974855789" watchObservedRunningTime="2026-09-16 19:50:08.451668035 +0000 UTC m=+4223.976126884"
Sep 16 19:50:10.272387 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:10.272352    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:10.378633 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:10.378600    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:24.825569 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:24.825530    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/0.log"
Sep 16 19:50:24.839302 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:24.839273    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/1.log"
Sep 16 19:50:24.876538 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:24.876507    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69d9584dd8-d72wj_cf98bdea-b950-4609-8382-89fc440c8bdc/router/0.log"
Sep 16 19:50:25.774314 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:25.774284    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/0.log"
Sep 16 19:50:25.791393 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:25.791362    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/1.log"
Sep 16 19:50:25.810421 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:25.810390    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69d9584dd8-d72wj_cf98bdea-b950-4609-8382-89fc440c8bdc/router/0.log"
Sep 16 19:50:26.889526 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:26.889488    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-sh65m_a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb/authorino/0.log"
Sep 16 19:50:27.045692 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:27.045665    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-s4vqj_5620cf6b-409b-4198-a9b7-0a4d0351c6b9/kuadrant-console-plugin/0.log"
Sep 16 19:50:32.529731 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:32.529693    2590 generic.go:356] "Generic (PLEG): container finished" podID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" containerID="7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8" exitCode=0
Sep 16 19:50:32.530221 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:32.529782    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svfwh/must-gather-wslk4" event={"ID":"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2","Type":"ContainerDied","Data":"7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8"}
Sep 16 19:50:32.530221 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:32.530136    2590 scope.go:117] "RemoveContainer" containerID="7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8"
Sep 16 19:50:32.715701 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:32.715667    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-svfwh_must-gather-wslk4_9c16b375-ece4-4a5d-9cbb-78eecad4c7e2/gather/0.log"
Sep 16 19:50:33.181580 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:33.181496    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:33.339205 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:33.339172    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:33.436423 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:33.436332    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:36.483425 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:36.483387    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qdskf_dcbf023b-195c-409b-b240-55139f258cb5/global-pull-secret-syncer/0.log"
Sep 16 19:50:36.558253 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:36.558221    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log"
Sep 16 19:50:36.734772 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:36.734689    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-44.ec2.internal_018261b2b77e73288a163ad4b4ee59f0/haproxy/0.log"
Sep 16 19:50:38.159266 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.159222    2590 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-svfwh/must-gather-wslk4"]
Sep 16 19:50:38.159754 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.159470    2590 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-svfwh/must-gather-wslk4" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" containerName="copy" containerID="cri-o://de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108" gracePeriod=2
Sep 16 19:50:38.165205 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:50:38.165154    2590 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-svfwh\"/\"must-gather-wslk4\": pods \"must-gather-wslk4\" not found" pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:38.165366 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.165281    2590 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-svfwh/must-gather-wslk4"]
Sep 16 19:50:38.387223 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.387201    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-svfwh_must-gather-wslk4_9c16b375-ece4-4a5d-9cbb-78eecad4c7e2/copy/0.log"
Sep 16 19:50:38.388698 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.388681    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:38.390492 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:50:38.390468    2590 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wslk4\" is forbidden: User \"system:node:ip-10-0-143-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-svfwh\": no relationship found between node 'ip-10-0-143-44.ec2.internal' and this object" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:38.422839 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.422768    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-must-gather-output\") pod \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\" (UID: \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\") "
Sep 16 19:50:38.422962 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.422862    2590 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wv9t\" (UniqueName: \"kubernetes.io/projected/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-kube-api-access-4wv9t\") pod \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\" (UID: \"9c16b375-ece4-4a5d-9cbb-78eecad4c7e2\") "
Sep 16 19:50:38.425361 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.425328    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-kube-api-access-4wv9t" (OuterVolumeSpecName: "kube-api-access-4wv9t") pod "9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" (UID: "9c16b375-ece4-4a5d-9cbb-78eecad4c7e2"). InnerVolumeSpecName "kube-api-access-4wv9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:38.429572 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.429550    2590 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" (UID: "9c16b375-ece4-4a5d-9cbb-78eecad4c7e2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:38.523482 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.523444    2590 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wv9t\" (UniqueName: \"kubernetes.io/projected/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-kube-api-access-4wv9t\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.523482 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.523476    2590 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2-must-gather-output\") on node \"ip-10-0-143-44.ec2.internal\" DevicePath \"\""
Sep 16 19:50:38.558189 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.558161    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-svfwh_must-gather-wslk4_9c16b375-ece4-4a5d-9cbb-78eecad4c7e2/copy/0.log"
Sep 16 19:50:38.558515 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.558488    2590 generic.go:356] "Generic (PLEG): container finished" podID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" containerID="de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108" exitCode=143
Sep 16 19:50:38.558618 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.558538    2590 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:38.558618 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.558566    2590 scope.go:117] "RemoveContainer" containerID="de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108"
Sep 16 19:50:38.560858 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:50:38.560832    2590 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wslk4\" is forbidden: User \"system:node:ip-10-0-143-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-svfwh\": no relationship found between node 'ip-10-0-143-44.ec2.internal' and this object" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:38.567118 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.567096    2590 scope.go:117] "RemoveContainer" containerID="7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8"
Sep 16 19:50:38.570095 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:50:38.570060    2590 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wslk4\" is forbidden: User \"system:node:ip-10-0-143-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-svfwh\": no relationship found between node 'ip-10-0-143-44.ec2.internal' and this object" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" pod="openshift-must-gather-svfwh/must-gather-wslk4"
Sep 16 19:50:38.581265 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.581245    2590 scope.go:117] "RemoveContainer" containerID="de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108"
Sep 16 19:50:38.581581 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:50:38.581558    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108\": container with ID starting with de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108 not found: ID does not exist" containerID="de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108"
Sep 16 19:50:38.581624 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.581589    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108"} err="failed to get container status \"de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108\": rpc error: code = NotFound desc = could not find container \"de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108\": container with ID starting with de5c697df87e5669dc3c0fd72c3e785610831a18122051036c965553e507a108 not found: ID does not exist"
Sep 16 19:50:38.581624 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.581609    2590 scope.go:117] "RemoveContainer" containerID="7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8"
Sep 16 19:50:38.581872 ip-10-0-143-44 kubenswrapper[2590]: E0916 19:50:38.581855    2590 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8\": container with ID starting with 7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8 not found: ID does not exist" containerID="7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8"
Sep 16 19:50:38.581924 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:38.581897    2590 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8"} err="failed to get container status \"7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8\": rpc error: code = NotFound desc = could not find container \"7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8\": container with ID starting with 7828785ab071e204cd5c8c0e2970da76267e279dba64b2425be52faa163535a8 not found: ID does not exist"
Sep 16 19:50:39.026649 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:39.026615    2590 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" path="/var/lib/kubelet/pods/9c16b375-ece4-4a5d-9cbb-78eecad4c7e2/volumes"
Sep 16 19:50:39.914362 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:39.914324    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:40.010930 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:40.010891    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:42.193456 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:42.193413    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-sh65m_a25c6e13-0dc9-47aa-a060-9cf2bb0e1ecb/authorino/0.log"
Sep 16 19:50:42.317746 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:42.317714    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-s4vqj_5620cf6b-409b-4198-a9b7-0a4d0351c6b9/kuadrant-console-plugin/0.log"
Sep 16 19:50:44.117534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:44.117504    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-6t2vh_32eb6be9-5836-434e-968e-623a8c1b4050/cluster-monitoring-operator/0.log"
Sep 16 19:50:44.514855 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:44.514778    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hhbjv_bb5a3fd8-458f-43ef-b9b9-75f1303197c1/node-exporter/0.log"
Sep 16 19:50:44.553202 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:44.553172    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hhbjv_bb5a3fd8-458f-43ef-b9b9-75f1303197c1/kube-rbac-proxy/0.log"
Sep 16 19:50:44.578829 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:44.578804    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hhbjv_bb5a3fd8-458f-43ef-b9b9-75f1303197c1/init-textfile/0.log"
Sep 16 19:50:46.976806 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.976770    2590 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"]
Sep 16 19:50:46.977222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.977051    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" containerName="gather"
Sep 16 19:50:46.977222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.977063    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" containerName="gather"
Sep 16 19:50:46.977222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.977072    2590 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" containerName="copy"
Sep 16 19:50:46.977222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.977078    2590 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" containerName="copy"
Sep 16 19:50:46.977222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.977132    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" containerName="gather"
Sep 16 19:50:46.977222 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.977142    2590 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c16b375-ece4-4a5d-9cbb-78eecad4c7e2" containerName="copy"
Sep 16 19:50:46.980724 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.980705    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:46.983030 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.983007    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s865n\"/\"kube-root-ca.crt\""
Sep 16 19:50:46.983159 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.983051    2590 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s865n\"/\"openshift-service-ca.crt\""
Sep 16 19:50:46.983706 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.983685    2590 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s865n\"/\"default-dockercfg-b7pzr\""
Sep 16 19:50:46.989635 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:46.989612    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"]
Sep 16 19:50:47.083741 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.083702    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-lib-modules\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.083968 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.083811    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-sys\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.083968 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.083850    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-proc\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.083968 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.083869    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtzg\" (UniqueName: \"kubernetes.io/projected/3cbb14da-af37-4844-8d4e-8b6fb57fd937-kube-api-access-5vtzg\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.083968 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.083911    2590 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-podres\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.184346 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.184307    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-lib-modules\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.184534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.184406    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-sys\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.184534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.184436    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-proc\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.184534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.184458    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtzg\" (UniqueName: \"kubernetes.io/projected/3cbb14da-af37-4844-8d4e-8b6fb57fd937-kube-api-access-5vtzg\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.184534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.184499    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-lib-modules\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.184534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.184515    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-proc\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.184534 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.184530    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-sys\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.184813 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.184526    2590 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-podres\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.184813 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.184626    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cbb14da-af37-4844-8d4e-8b6fb57fd937-podres\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.193828 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.193799    2590 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtzg\" (UniqueName: \"kubernetes.io/projected/3cbb14da-af37-4844-8d4e-8b6fb57fd937-kube-api-access-5vtzg\") pod \"perf-node-gather-daemonset-qd9pn\" (UID: \"3cbb14da-af37-4844-8d4e-8b6fb57fd937\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.291205 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.291171    2590 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:47.560601 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.560491    2590 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"]
Sep 16 19:50:47.563422 ip-10-0-143-44 kubenswrapper[2590]: W0916 19:50:47.563395    2590 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3cbb14da_af37_4844_8d4e_8b6fb57fd937.slice/crio-f28d4e0c0fab8c1ae6ef9144f41fd30e1cd713f573e36fc619fe54577ccf74a1 WatchSource:0}: Error finding container f28d4e0c0fab8c1ae6ef9144f41fd30e1cd713f573e36fc619fe54577ccf74a1: Status 404 returned error can't find the container with id f28d4e0c0fab8c1ae6ef9144f41fd30e1cd713f573e36fc619fe54577ccf74a1
Sep 16 19:50:47.600852 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:47.600818    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn" event={"ID":"3cbb14da-af37-4844-8d4e-8b6fb57fd937","Type":"ContainerStarted","Data":"f28d4e0c0fab8c1ae6ef9144f41fd30e1cd713f573e36fc619fe54577ccf74a1"}
Sep 16 19:50:48.606079 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:48.606044    2590 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn" event={"ID":"3cbb14da-af37-4844-8d4e-8b6fb57fd937","Type":"ContainerStarted","Data":"61e200f4c6d70bb6f769666300649740460601e282d11d283e30520002304a31"}
Sep 16 19:50:48.606485 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:48.606147    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:48.624293 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:48.624244    2590 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn" podStartSLOduration=2.624224372 podStartE2EDuration="2.624224372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:48.623402846 +0000 UTC m=+4264.147861680" watchObservedRunningTime="2026-09-16 19:50:48.624224372 +0000 UTC m=+4264.148683250"
Sep 16 19:50:48.969638 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:48.969560    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7hhn9_5a09e3f9-b797-41a8-9929-4f5c73ebab63/dns/0.log"
Sep 16 19:50:48.991390 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:48.991361    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7hhn9_5a09e3f9-b797-41a8-9929-4f5c73ebab63/kube-rbac-proxy/0.log"
Sep 16 19:50:49.131471 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:49.131432    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nwx8j_c752cc3d-aa4e-477a-8347-8e2820392135/dns-node-resolver/0.log"
Sep 16 19:50:49.774393 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:49.774363    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9gwts_c12ef66b-21e1-431d-abde-fc6f7987622a/node-ca/0.log"
Sep 16 19:50:50.866113 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:50.866072    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:50.931614 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:50.931564    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/0.log"
Sep 16 19:50:50.945192 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:50.945159    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/1.log"
Sep 16 19:50:50.950158 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:50.950131    2590 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kn9bc_796bc1bc-020c-43c9-86ee-ba72e89d5b11/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:50.999762 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:50.999729    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69d9584dd8-d72wj_cf98bdea-b950-4609-8382-89fc440c8bdc/router/0.log"
Sep 16 19:50:51.567657 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:51.567629    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cjk4p_bb9df607-c5c5-41b9-86a0-45503173194f/serve-healthcheck-canary/0.log"
Sep 16 19:50:52.161867 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:52.161828    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-h6hs2_eb13daed-c369-49b7-9e38-bdd0636bd0ad/insights-operator/0.log"
Sep 16 19:50:52.162930 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:52.162912    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-h6hs2_eb13daed-c369-49b7-9e38-bdd0636bd0ad/insights-operator/1.log"
Sep 16 19:50:52.271867 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:52.271835    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tq2g6_73588ab8-b492-432b-953e-68467d462260/kube-rbac-proxy/0.log"
Sep 16 19:50:52.293478 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:52.293451    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tq2g6_73588ab8-b492-432b-953e-68467d462260/exporter/0.log"
Sep 16 19:50:52.314691 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:52.314660    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tq2g6_73588ab8-b492-432b-953e-68467d462260/extractor/0.log"
Sep 16 19:50:54.621436 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:54.621407    2590 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-qd9pn"
Sep 16 19:50:55.521570 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:55.521536    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-856c96b694-9x9nz_d76c8eb5-fb50-45f4-a2b5-8dfb0804ca35/manager/0.log"
Sep 16 19:50:55.576092 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:55.576066    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-h5vpr_72131bdb-e31a-4477-bbaa-694c991657fd/openshift-lws-operator/0.log"
Sep 16 19:50:57.063462 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:57.063430    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/0.log"
Sep 16 19:50:57.084440 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:57.084396    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/1.log"
Sep 16 19:50:57.116149 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:57.116019    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69d9584dd8-d72wj_cf98bdea-b950-4609-8382-89fc440c8bdc/router/0.log"
Sep 16 19:50:58.174112 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:58.174040    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qf4tq_e464a8d1-7497-4f4e-b1e2-2ec875216a5e/s3-init/0.log"
Sep 16 19:50:58.190618 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:58.190589    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qf4tq_e464a8d1-7497-4f4e-b1e2-2ec875216a5e/download-hf-model/0.log"
Sep 16 19:50:58.463724 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:58.463639    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/0.log"
Sep 16 19:50:58.478254 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:58.478220    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-blgj4_a9490cec-b84b-4cc7-aefa-946d4e5646fb/istio-proxy/1.log"
Sep 16 19:50:58.495513 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:50:58.495489    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69d9584dd8-d72wj_cf98bdea-b950-4609-8382-89fc440c8bdc/router/0.log"
Sep 16 19:51:03.436414 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:03.436386    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5lrj2_5da5b179-92bd-4308-a524-a08566a60d2d/kube-storage-version-migrator-operator/1.log"
Sep 16 19:51:03.437249 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:03.437234    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5lrj2_5da5b179-92bd-4308-a524-a08566a60d2d/kube-storage-version-migrator-operator/0.log"
Sep 16 19:51:04.825270 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:04.825230    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swzz6_191f585d-d90c-4b3c-8792-8a0d4f6abb19/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:51:04.846560 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:04.846528    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swzz6_191f585d-d90c-4b3c-8792-8a0d4f6abb19/egress-router-binary-copy/0.log"
Sep 16 19:51:04.870328 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:04.870300    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swzz6_191f585d-d90c-4b3c-8792-8a0d4f6abb19/cni-plugins/0.log"
Sep 16 19:51:04.892396 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:04.892373    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swzz6_191f585d-d90c-4b3c-8792-8a0d4f6abb19/bond-cni-plugin/0.log"
Sep 16 19:51:04.914033 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:04.914002    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swzz6_191f585d-d90c-4b3c-8792-8a0d4f6abb19/routeoverride-cni/0.log"
Sep 16 19:51:04.939238 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:04.939214    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swzz6_191f585d-d90c-4b3c-8792-8a0d4f6abb19/whereabouts-cni-bincopy/0.log"
Sep 16 19:51:04.965123 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:04.965090    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swzz6_191f585d-d90c-4b3c-8792-8a0d4f6abb19/whereabouts-cni/0.log"
Sep 16 19:51:05.005233 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:05.005204    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bvhwm_a91166ef-fb72-4152-87d9-7f6b9d7a127b/kube-multus/0.log"
Sep 16 19:51:05.155567 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:05.155478    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pg5zg_5989f247-505b-4069-92dc-8d563bfeaae6/network-metrics-daemon/0.log"
Sep 16 19:51:05.174964 ip-10-0-143-44 kubenswrapper[2590]: I0916 19:51:05.174935    2590 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pg5zg_5989f247-505b-4069-92dc-8d563bfeaae6/kube-rbac-proxy/0.log"