Sep 16 18:43:39.619228 ip-10-0-139-95 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:43:40.029640 ip-10-0-139-95 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:40.029640 ip-10-0-139-95 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:43:40.029640 ip-10-0-139-95 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:40.029640 ip-10-0-139-95 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:43:40.029640 ip-10-0-139-95 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:40.030440 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.029670    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:43:40.034511 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034488    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:40.034511 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034508    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:40.034511 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034513    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:40.034511 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034518    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034524    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034529    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034533    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034537    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034541    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034544    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034548    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034552    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034556    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034560    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034564    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034568    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034571    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034575    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034579    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034582    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034586    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034591    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034595    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:40.034763 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034599    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034602    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034607    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034611    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034615    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034620    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034623    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034627    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034635    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034640    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034646    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034651    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034655    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034660    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034667    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034671    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034675    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034680    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034684    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034688    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:40.035603 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034692    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034696    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034700    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034704    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034708    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034712    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034717    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034721    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034725    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034729    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034733    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034741    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034746    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034751    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034755    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034759    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034764    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034768    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034772    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:40.036459 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034775    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034780    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034785    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034789    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034794    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034797    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034802    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034810    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034816    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034822    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034826    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034831    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034835    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034839    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034843    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034848    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034852    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034856    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034860    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:40.037275 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034864    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034868    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034872    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034877    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034882    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034885    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034912    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034917    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034921    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034926    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034930    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034934    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034938    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034943    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034947    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034951    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034955    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.034959    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036129    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036140    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:40.038169 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036144    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036149    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036153    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036159    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036163    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036167    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036171    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036175    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036179    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036184    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036188    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036192    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036196    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036200    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036204    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036208    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036213    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036217    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036222    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:40.039054 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036226    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036230    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036234    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036238    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036242    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036246    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036250    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036254    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036259    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036263    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036267    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036271    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036276    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036280    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036284    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036288    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036292    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036297    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036302    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036306    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:40.039661 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036311    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036316    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036320    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036325    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036329    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036333    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036336    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036341    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036345    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036353    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036358    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036363    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036367    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036372    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036376    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036380    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036385    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036389    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036393    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036397    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:40.040287 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036401    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036405    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036409    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036413    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036417    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036421    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036425    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036430    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036434    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036438    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036443    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036447    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036451    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036455    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036459    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036465    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036470    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036477    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036482    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:40.040814 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036487    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036492    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036498    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036503    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036511    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036516    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036521    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036526    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036531    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036536    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036541    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036546    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036551    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036555    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036559    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036564    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036569    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036573    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.036578    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:40.041402 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037217    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037227    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037231    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037235    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037240    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037246    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037250    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037254    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037259    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037263    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037267    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037271    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037274    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037278    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037283    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037289    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037293    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037298    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037302    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037306    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:40.042064 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037310    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037314    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037319    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037323    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037327    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037332    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037336    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037340    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037344    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037351    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037356    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037361    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037366    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037370    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037374    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037380    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037384    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037393    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037398    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:40.042559 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037403    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037407    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037412    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037416    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037420    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037424    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037428    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037432    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037438    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037442    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037446    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037450    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037454    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037458    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037462    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037466    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037470    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037474    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037478    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037482    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:40.043142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037486    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037490    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037494    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037498    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037501    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037505    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037509    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037513    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037517    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037521    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037525    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037531    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037537    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037541    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037545    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037549    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037553    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037557    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037560    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037564    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:40.043630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037570    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037574    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037580    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037585    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037590    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037593    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037598    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037602    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037606    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037610    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037615    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037618    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037623    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037630    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037636    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037641    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037646    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037651    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037656    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:40.044181 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.037660    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037771    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037836    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037848    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037854    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037864    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037870    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037876    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037883    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037908    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037913    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037927    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037932    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037939    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037944    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037948    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037953    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037958    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037963    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037970    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037974    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037979    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037984    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037990    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:43:40.044646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.037995    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038001    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038006    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038011    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038016    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038021    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038027    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038032    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038036    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038044    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038049    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038055    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038060    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038067    2580 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038073    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038080    2580 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038086    2580 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038091    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038095    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038100    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038106    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038113    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038117    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038122    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038127    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:43:40.045247 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038132    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038136    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038141    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038145    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038150    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038154    2580 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038161    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038165    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038170    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038175    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038179    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038184    2580 flags.go:64] FLAG: --help="false"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038188    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038193    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038198    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038203    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038208    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038214    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038219    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038223    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038229    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038234    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038240    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038245    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:43:40.045884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038250    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038255    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038259    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038265    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038270    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038275    2580 flags.go:64] FLAG: --lock-file=""
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038279    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038284    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038288    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038297    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038301    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038306    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038310    2580 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038315    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038320    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038345    2580 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038352    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038359    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038364    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038370    2580 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038375    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038380    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038385    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038389    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038394    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:43:40.046916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038399    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038404    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038416    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038423    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038427    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038432    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038438    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038447    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038451    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038456    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038461    2580 flags.go:64] FLAG: --port="10250"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038466    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038470    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088f62d861ec4a759"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038475    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038480    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038484    2580 flags.go:64] FLAG: --register-node="true"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038489    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038495    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038499    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038504    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038508    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038514    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038519    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038523    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038528    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:43:40.047999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038533    2580 flags.go:64] FLAG: --runonce="false"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038537    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038542    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038547    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038551    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038557    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038561    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038566    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038571    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038576    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038582    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038586    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038591    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038595    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038600    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038604    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038613    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038617    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038621    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038628    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038632    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038637    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038641    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038646    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038651    2580 flags.go:64] FLAG: --v="2"
Sep 16 18:43:40.048873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038657    2580 flags.go:64] FLAG: --version="false"
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038664    2580 flags.go:64] FLAG: --vmodule=""
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038670    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.038675    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038866    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038873    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038879    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038884    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038906    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038911    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038915    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038920    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038924    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038928    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038931    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038935    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038940    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038949    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038954    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038959    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:40.049750 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038963    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038967    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038973    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038977    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038982    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038986    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038990    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038994    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.038998    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039002    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039009    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039015    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039021    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039027    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039032    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039037    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039042    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039047    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039051    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:40.050633 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039055    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039059    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039063    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039067    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039070    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039075    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039079    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039082    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039090    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039094    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039101    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039105    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039110    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039114    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039118    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039122    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039128    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039133    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039137    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039141    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:40.051277 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039145    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039149    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039153    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039157    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039161    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039165    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039169    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039174    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039178    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039182    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039186    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039191    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039194    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039199    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039203    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039206    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039210    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039215    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039219    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039223    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:40.052142 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039229    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039233    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039239    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039243    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039247    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039251    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039255    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039259    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039263    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039268    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039273    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039278    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039282    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039286    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039291    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039295    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039299    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039303    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039307    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039311    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:40.052675 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039315    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039320    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039323    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.039327    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.039944    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.048382    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.048400    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.049227    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.049244    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049349    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049358    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049363    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049368    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049373    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049378    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:40.053299 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049382    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049387    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049391    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049396    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049399    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049403    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049407    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049411    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049415    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049419    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049423    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049427    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049431    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049435    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049439    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049443    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049447    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049451    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049455    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049459    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:40.053832 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049466    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049472    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049476    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049482    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049487    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049491    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049495    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049500    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049504    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049509    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049513    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049518    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049522    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049527    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049531    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049536    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049539    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049543    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049548    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049552    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:40.054445 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049556    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049561    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049565    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049570    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049574    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049578    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049584    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049590    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049594    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049598    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049602    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049606    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049611    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049615    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049619    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049623    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049628    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049633    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049637    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049642    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:40.055109 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049646    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049650    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049657    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049662    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049667    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049671    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049676    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049680    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049684    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049688    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049692    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049696    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049701    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049705    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049709    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049713    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049717    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049722    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049725    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:40.055606 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049730    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049734    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049738    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049742    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049746    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049749    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049753    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049757    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049761    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049765    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049769    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049773    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049778    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.049782    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:40.056151 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.049790    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050009    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050020    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050026    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050030    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050035    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050039    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050043    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050047    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050051    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050055    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050059    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050064    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050068    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050072    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050076    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050081    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050085    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050089    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050093    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:40.056501 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050097    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050100    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050104    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050108    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050113    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050117    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050121    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050125    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050129    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050135    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050139    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050143    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050147    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050152    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050156    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050160    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050164    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050168    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050173    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050177    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:40.057069 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050181    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050184    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050188    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050192    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050197    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050201    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050205    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050210    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050214    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050218    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050222    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050226    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050230    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050234    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050238    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050243    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050246    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050250    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050254    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050258    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:40.057577 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050263    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050267    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050271    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050275    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050279    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050283    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050287    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050291    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050296    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050300    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050304    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050308    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050312    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050316    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050323    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050327    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050332    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050338    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050344    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:40.058103 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050349    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050354    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050360    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050364    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050369    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050373    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050377    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050381    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050388    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050393    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050398    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050403    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050407    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050412    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050416    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050420    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050426    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050431    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050435    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050439    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:40.058806 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:40.050443    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:40.059646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.050451    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:40.059646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.050615    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:43:40.059646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.054639    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:43:40.059646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.055806    2580 server.go:1019] "Starting client certificate rotation"
Sep 16 18:43:40.059646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.055924    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:40.059646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.055967    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:40.059646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.058958    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:40.061057 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.061021    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:40.072966 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.072876    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:43:40.076566 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.076552    2580 log.go:25] "Validated CRI v1 image API"
Sep 16 18:43:40.077445 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.077432    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:43:40.082601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.082583    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d8d964f2-9247-4247-af72-087b0d3d0c36:/dev/nvme0n1p4 f01a5164-c0f4-492a-8d76-dda548b0e94e:/dev/nvme0n1p3]
Sep 16 18:43:40.082648 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.082601    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:43:40.086845 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.086819    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:40.087786 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.087678    2580 manager.go:216] Machine: {Timestamp:2026-09-16 18:43:40.086412462 +0000 UTC m=+0.348089465 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126563 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21a0fa55daa0af0c4083c474b431fa SystemUUID:ec21a0fa-55da-a0af-0c40-83c474b431fa BootID:96cbf605-4771-4be0-8572-7018f05e8c75 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:1f:4b:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:1f:4b:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:ea:82:f0:19:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:43:40.087786 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.087784    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:43:40.087964 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.087922    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:43:40.091398 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.091374    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:43:40.091536 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.091401    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-95.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:43:40.092736 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.092726    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:43:40.092770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.092740    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:43:40.092770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.092748    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:40.093466 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.093456    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:40.094321 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.094311    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:43:40.098391 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.098379    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:40.098513 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.098504    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:43:40.099191 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.099180    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:43:40.099252 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.099201    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:43:40.099252 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.099216    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:43:40.099252 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.099225    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:43:40.099252 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.099235    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:43:40.099613 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.099602    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:40.099660 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.099621    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:40.102153 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.102139    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:43:40.104739 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.104724    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:43:40.104781 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.104756    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:43:40.106422 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106411    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:43:40.106464 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106428    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:43:40.106464 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106435    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:43:40.106464 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106440    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:43:40.106464 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106448    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:43:40.106464 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106454    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:43:40.106597 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106468    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:43:40.106597 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106473    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:43:40.106597 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106480    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:43:40.106597 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106485    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:43:40.106597 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106499    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:43:40.106597 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.106505    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:43:40.107254 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.107243    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:43:40.107284 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.107256    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:43:40.109660 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.109643    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-95.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:43:40.109708 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.109691    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:43:40.109742 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.109680    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:43:40.111241 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.111229    2580 server.go:1268] "Started kubelet"
Sep 16 18:43:40.111326 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.111296    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:43:40.111380 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.111331    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:43:40.111428 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.111412    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:43:40.112099 ip-10-0-139-95 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:43:40.112771 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.112656    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:43:40.116142 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.116124    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:43:40.117978 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.117950    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jmpzg"
Sep 16 18:43:40.118369 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.118327    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:43:40.120779 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.120763    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:40.121411 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.121399    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:43:40.121607 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.121477    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.122052 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122030    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:43:40.122128 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122033    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:43:40.122128 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122068    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:43:40.122234 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122157    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:43:40.122234 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122165    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:43:40.122663 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122649    2580 factory.go:153] Registering CRI-O factory
Sep 16 18:43:40.122730 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122666    2580 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:43:40.122730 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122719    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:43:40.122730 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122729    2580 factory.go:55] Registering systemd factory
Sep 16 18:43:40.122864 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122736    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:43:40.122864 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122751    2580 factory.go:103] Registering Raw factory
Sep 16 18:43:40.122864 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.122765    2580 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:43:40.123138 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.123124    2580 manager.go:349] Starting recovery of all containers
Sep 16 18:43:40.123695 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.123678    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jmpzg"
Sep 16 18:43:40.131285 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.131261    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:40.133431 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.133413    2580 manager.go:354] Recovery completed
Sep 16 18:43:40.135366 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.135333    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-95.ec2.internal\" not found" node="ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.138099 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.138085    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:40.140647 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.140628    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:40.140734 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.140658    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:40.140734 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.140669    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:40.141434 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.141419    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:43:40.141434 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.141432    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:40.141520 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.141435    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:43:40.141520 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.141455    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:40.142324 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.142308    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:40.142388 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.142340    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:40.142388 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.142351    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:40.143524 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.143504    2580 policy_none.go:49] "None policy: Start"
Sep 16 18:43:40.143524 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.143522    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:43:40.143671 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.143532    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:43:40.145608 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.145596    2580 policy_none.go:47] "Start"
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.182773    2580 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.182810    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.182823    2580 server.go:87] "Starting device plugin registration server"
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.183096    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.183106    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.183202    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.183289    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.183301    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.183981    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:43:40.202604 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.184036    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.267957 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.267740    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:43:40.269273 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.269252    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:43:40.269398 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.269314    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:43:40.269398 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.269343    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:43:40.269510 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.269455    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:43:40.271904 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.271868    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:40.283923 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.283849    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:40.284764 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.284748    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:40.284841 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.284782    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:40.284841 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.284797    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:40.284841 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.284827    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.294351 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.294329    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.294443 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.294358    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-95.ec2.internal\": node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.314965 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.314935    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.370098 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.370044    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal"]
Sep 16 18:43:40.370235 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.370163    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:40.371818 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.371797    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:40.371960 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.371831    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:40.371960 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.371847    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:40.374047 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.374033    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:40.374154 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.374139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.374229 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.374169    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:40.375764 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.375535    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:40.375764 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.375563    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:40.375764 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.375576    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:40.375960 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.375833    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:40.375960 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.375850    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:40.375960 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.375862    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:40.378032 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.378008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.378119 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.378036    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:40.378742 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.378726    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:40.378793 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.378755    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:40.378793 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.378770    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:40.407582 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.407559    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-95.ec2.internal\" not found" node="ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.411809 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.411793    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-95.ec2.internal\" not found" node="ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.415844 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.415829    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.423695 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.423680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/94f8490c5f2548f89f7eaeda969f1633-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal\" (UID: \"94f8490c5f2548f89f7eaeda969f1633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.423761 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.423702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f8490c5f2548f89f7eaeda969f1633-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal\" (UID: \"94f8490c5f2548f89f7eaeda969f1633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.423761 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.423721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c92465bf8a81cf25e8e179fc079decff-config\") pod \"kube-apiserver-proxy-ip-10-0-139-95.ec2.internal\" (UID: \"c92465bf8a81cf25e8e179fc079decff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.516232 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.516196    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.524824 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.524788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/94f8490c5f2548f89f7eaeda969f1633-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal\" (UID: \"94f8490c5f2548f89f7eaeda969f1633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.524979 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.524834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f8490c5f2548f89f7eaeda969f1633-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal\" (UID: \"94f8490c5f2548f89f7eaeda969f1633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.524979 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.524878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c92465bf8a81cf25e8e179fc079decff-config\") pod \"kube-apiserver-proxy-ip-10-0-139-95.ec2.internal\" (UID: \"c92465bf8a81cf25e8e179fc079decff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.524979 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.524931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f8490c5f2548f89f7eaeda969f1633-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal\" (UID: \"94f8490c5f2548f89f7eaeda969f1633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.524979 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.524936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c92465bf8a81cf25e8e179fc079decff-config\") pod \"kube-apiserver-proxy-ip-10-0-139-95.ec2.internal\" (UID: \"c92465bf8a81cf25e8e179fc079decff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.524979 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.524931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/94f8490c5f2548f89f7eaeda969f1633-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal\" (UID: \"94f8490c5f2548f89f7eaeda969f1633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.616858 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.616784    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.710386 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.710347    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.714128 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.714105    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:40.717353 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.717329    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.818216 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.818178    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.918793 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:40.918676    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:40.951619 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:40.951595    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:41.019089 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:41.019054    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:41.055545 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.055517    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:43:41.056156 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.055673    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:43:41.056156 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.055673    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:43:41.056156 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.055702    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:43:41.119837 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:41.119806    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:41.127851 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.127821    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:38:40 +0000 UTC" deadline="2028-04-03 16:40:35.018302618 +0000 UTC"
Sep 16 18:43:41.127851 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.127847    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13557h56m53.890459221s"
Sep 16 18:43:41.220093 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:41.220057    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:41.321660 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:41.321619    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:41.347782 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:41.347749    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f8490c5f2548f89f7eaeda969f1633.slice/crio-0fc569f104182f8b90d75b8c896c0fa324cf99d71e8f249d93523548aa231cec WatchSource:0}: Error finding container 0fc569f104182f8b90d75b8c896c0fa324cf99d71e8f249d93523548aa231cec: Status 404 returned error can't find the container with id 0fc569f104182f8b90d75b8c896c0fa324cf99d71e8f249d93523548aa231cec
Sep 16 18:43:41.348066 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:41.348048    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92465bf8a81cf25e8e179fc079decff.slice/crio-f6d03c8a37608215265ac4449356f2e18ce849c7dbf77a620035c7196d0d7e64 WatchSource:0}: Error finding container f6d03c8a37608215265ac4449356f2e18ce849c7dbf77a620035c7196d0d7e64: Status 404 returned error can't find the container with id f6d03c8a37608215265ac4449356f2e18ce849c7dbf77a620035c7196d0d7e64
Sep 16 18:43:41.353115 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.353096    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:43:41.422459 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:41.422418    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:41.528965 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:41.528872    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:41.630113 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:41.630075    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-95.ec2.internal\" not found"
Sep 16 18:43:41.668144 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.668118    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:41.722599 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.722558    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:41.734762 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.734737    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:41.735811 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.735797    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal"
Sep 16 18:43:41.743785 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:41.743765    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:42.029473 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.029429    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:42.100580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.100283    2580 apiserver.go:52] "Watching apiserver"
Sep 16 18:43:42.106354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.106300    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:43:42.107676 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.107626    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-4cwkd","kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal","openshift-dns/node-resolver-x9lhv","openshift-image-registry/node-ca-4t8hn","openshift-multus/multus-7nxf5","openshift-multus/multus-additional-cni-plugins-j8qqv","openshift-network-diagnostics/network-check-target-rr9t2","openshift-ovn-kubernetes/ovnkube-node-6dwh7","kube-system/konnectivity-agent-wwmwd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht","openshift-cluster-node-tuning-operator/tuned-tsbtz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal","openshift-multus/network-metrics-daemon-knv9v","openshift-network-operator/iptables-alerter-bfg62"]
Sep 16 18:43:42.110268 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.110234    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:42.110374 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.110308    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:43:42.112883 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.112856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:42.113005 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.112950    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:43:42.115587 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.115560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.118218 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.118195    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rx2ts\""
Sep 16 18:43:42.118460 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.118432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.119501 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.119480    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:43:42.120370 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.120347    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:43:42.120519 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.120496    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:43:42.120635 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.120609    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:43:42.120746 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.120716    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:43:42.120837 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.120817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.120902 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.120825    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lljrs\""
Sep 16 18:43:42.121463 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.121439    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:42.122896 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.122879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.123649 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.123619    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:43:42.123744 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.123679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7ksft\""
Sep 16 18:43:42.123815 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.123725    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:43:42.123815 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.123679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:43:42.123921 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.123901    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:43:42.125268 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.125252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:43:42.126031 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.126004    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fh4l5\""
Sep 16 18:43:42.126285 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.126262    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:43:42.126472 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.126455    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:43:42.126759 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.126727    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:42.127190 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.127143    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:43:42.127438 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.127420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7lj9r\""
Sep 16 18:43:42.127632 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.127615    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:43:42.129538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.129513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.129646 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.129590    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:43:42.132447 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.132430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.133433 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.133412    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:42.134493 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.134473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-run-k8s-cni-cncf-io\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.135159 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.135131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c885636-ca48-468f-b59b-a70323e7e02e-cnibin\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.135333 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.135314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh7dj\" (UniqueName: \"kubernetes.io/projected/2c885636-ca48-468f-b59b-a70323e7e02e-kube-api-access-fh7dj\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.135459 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.134898    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:42.135459 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.135033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.135668 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.135646    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:43:42.135765 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.134819    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxf5q\""
Sep 16 18:43:42.135838 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.134881    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:42.135930 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.135428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-hostroot\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.135930 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.135875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flkmm\" (UniqueName: \"kubernetes.io/projected/3621c163-c9d0-4de5-9e47-e58b7af3729d-kube-api-access-flkmm\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.135930 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.135912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-conf-dir\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.136085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.135968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzxmg\" (UniqueName: \"kubernetes.io/projected/3e740f22-4534-4351-8405-41938e2ad52c-kube-api-access-vzxmg\") pod \"node-resolver-x9lhv\" (UID: \"3e740f22-4534-4351-8405-41938e2ad52c\") " pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.136085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.135997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-cnibin\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.136085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-socket-dir-parent\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.136085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-run-netns\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.136085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c198208c-5766-492e-8ae7-40c2ea1a637c-host\") pod \"node-ca-4t8hn\" (UID: \"c198208c-5766-492e-8ae7-40c2ea1a637c\") " pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.136283 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-var-lib-cni-multus\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.136283 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c885636-ca48-468f-b59b-a70323e7e02e-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.136283 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c885636-ca48-468f-b59b-a70323e7e02e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.136283 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-etc-kubernetes\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.136283 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c885636-ca48-468f-b59b-a70323e7e02e-system-cni-dir\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.136283 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c885636-ca48-468f-b59b-a70323e7e02e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.136283 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e740f22-4534-4351-8405-41938e2ad52c-tmp-dir\") pod \"node-resolver-x9lhv\" (UID: \"3e740f22-4534-4351-8405-41938e2ad52c\") " pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c198208c-5766-492e-8ae7-40c2ea1a637c-serviceca\") pod \"node-ca-4t8hn\" (UID: \"c198208c-5766-492e-8ae7-40c2ea1a637c\") " pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvff\" (UniqueName: \"kubernetes.io/projected/c198208c-5766-492e-8ae7-40c2ea1a637c-kube-api-access-cpvff\") pod \"node-ca-4t8hn\" (UID: \"c198208c-5766-492e-8ae7-40c2ea1a637c\") " pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-cni-dir\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-var-lib-kubelet\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-daemon-config\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkdn\" (UniqueName: \"kubernetes.io/projected/67d8b168-67d4-4357-a091-52075e672f5f-kube-api-access-5tkdn\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c885636-ca48-468f-b59b-a70323e7e02e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.136625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-system-cni-dir\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.137034 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3621c163-c9d0-4de5-9e47-e58b7af3729d-cni-binary-copy\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.137034 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-var-lib-cni-bin\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.137034 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-run-multus-certs\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.137034 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.136727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e740f22-4534-4351-8405-41938e2ad52c-hosts-file\") pod \"node-resolver-x9lhv\" (UID: \"3e740f22-4534-4351-8405-41938e2ad52c\") " pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.137789 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.137766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:43:42.137885 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.137823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-twbgl\""
Sep 16 18:43:42.137953 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.137770    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:43:42.138072 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.138057    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:43:42.138667 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.138646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.140889 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.140869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttpm5\""
Sep 16 18:43:42.141948 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.141470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.141948 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.141661    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:42.143102 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.143024    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:42.151488 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.151466    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:43:42.151488 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.151484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ql55z\""
Sep 16 18:43:42.151793 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.151773    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:43:42.152097 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.152079    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:43:42.152184 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.152170    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:43:42.152312 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.152081    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:43:42.152378 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.152321    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:43:42.160654 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.160633    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dftzw"
Sep 16 18:43:42.170196 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.170176    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dftzw"
Sep 16 18:43:42.223674 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.223648    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:43:42.237540 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-kubelet\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.237667 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7a87c4-b91c-491e-9c5c-deab72585f4d-kubelet-config\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.237667 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-lib-modules\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.237667 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e83e603-da27-4035-888a-811cc80ca89f-tmp\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.237667 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-etc-kubernetes\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.237836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c885636-ca48-468f-b59b-a70323e7e02e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.237836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvff\" (UniqueName: \"kubernetes.io/projected/c198208c-5766-492e-8ae7-40c2ea1a637c-kube-api-access-cpvff\") pod \"node-ca-4t8hn\" (UID: \"c198208c-5766-492e-8ae7-40c2ea1a637c\") " pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.237836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-sysctl-d\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.237836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-slash\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.237836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-log-socket\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.237836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-cni-bin\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.237836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-cni-dir\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.237836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkdn\" (UniqueName: \"kubernetes.io/projected/67d8b168-67d4-4357-a091-52075e672f5f-kube-api-access-5tkdn\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:42.237836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfd5\" (UniqueName: \"kubernetes.io/projected/db3f5719-c1f7-40ea-83fb-395aaca843e4-kube-api-access-zcfd5\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-run-netns\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-run-systemd\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-etc-openvswitch\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-node-log\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c885636-ca48-468f-b59b-a70323e7e02e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.237992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-run-multus-certs\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbe1e74d-fc0c-48b5-aedc-a3822f634fa5-iptables-alerter-script\") pod \"iptables-alerter-bfg62\" (UID: \"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5\") " pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbkq\" (UniqueName: \"kubernetes.io/projected/dbe1e74d-fc0c-48b5-aedc-a3822f634fa5-kube-api-access-rhbkq\") pod \"iptables-alerter-bfg62\" (UID: \"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5\") " pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-run-ovn-kubernetes\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-run-k8s-cni-cncf-io\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c885636-ca48-468f-b59b-a70323e7e02e-cnibin\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-run\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-run-ovn\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-cni-netd\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-modprobe-d\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.238360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-sysconfig\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbe1e74d-fc0c-48b5-aedc-a3822f634fa5-host-slash\") pod \"iptables-alerter-bfg62\" (UID: \"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5\") " pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-var-lib-openvswitch\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e9179b5-f018-4459-bd59-5c2600534509-ovnkube-config\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-cnibin\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-var-lib-cni-multus\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c885636-ca48-468f-b59b-a70323e7e02e-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c885636-ca48-468f-b59b-a70323e7e02e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c885636-ca48-468f-b59b-a70323e7e02e-system-cni-dir\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e740f22-4534-4351-8405-41938e2ad52c-tmp-dir\") pod \"node-resolver-x9lhv\" (UID: \"3e740f22-4534-4351-8405-41938e2ad52c\") " pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c198208c-5766-492e-8ae7-40c2ea1a637c-serviceca\") pod \"node-ca-4t8hn\" (UID: \"c198208c-5766-492e-8ae7-40c2ea1a637c\") " pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-systemd\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z8np\" (UniqueName: \"kubernetes.io/projected/8e83e603-da27-4035-888a-811cc80ca89f-kube-api-access-8z8np\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e9179b5-f018-4459-bd59-5c2600534509-env-overrides\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-var-lib-kubelet\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-daemon-config\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.239354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-registration-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-systemd-units\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-system-cni-dir\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3621c163-c9d0-4de5-9e47-e58b7af3729d-cni-binary-copy\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-var-lib-cni-bin\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e740f22-4534-4351-8405-41938e2ad52c-hosts-file\") pod \"node-resolver-x9lhv\" (UID: \"3e740f22-4534-4351-8405-41938e2ad52c\") " pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bff8a-35f8-4491-a764-3e1318d4b95f-konnectivity-ca\") pod \"konnectivity-agent-wwmwd\" (UID: \"cb9bff8a-35f8-4491-a764-3e1318d4b95f\") " pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-sys\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-socket-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-sys-fs\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh7dj\" (UniqueName: \"kubernetes.io/projected/2c885636-ca48-468f-b59b-a70323e7e02e-kube-api-access-fh7dj\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-kubernetes\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-host\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e83e603-da27-4035-888a-811cc80ca89f-etc-tuned\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.239961 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-run-openvswitch\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e9179b5-f018-4459-bd59-5c2600534509-ovnkube-script-lib\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltm2m\" (UniqueName: \"kubernetes.io/projected/9e9179b5-f018-4459-bd59-5c2600534509-kube-api-access-ltm2m\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-hostroot\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flkmm\" (UniqueName: \"kubernetes.io/projected/3621c163-c9d0-4de5-9e47-e58b7af3729d-kube-api-access-flkmm\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.238998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-conf-dir\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzxmg\" (UniqueName: \"kubernetes.io/projected/3e740f22-4534-4351-8405-41938e2ad52c-kube-api-access-vzxmg\") pod \"node-resolver-x9lhv\" (UID: \"3e740f22-4534-4351-8405-41938e2ad52c\") " pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb9bff8a-35f8-4491-a764-3e1318d4b95f-agent-certs\") pod \"konnectivity-agent-wwmwd\" (UID: \"cb9bff8a-35f8-4491-a764-3e1318d4b95f\") " pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-etc-kubernetes\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-var-lib-cni-bin\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e740f22-4534-4351-8405-41938e2ad52c-hosts-file\") pod \"node-resolver-x9lhv\" (UID: \"3e740f22-4534-4351-8405-41938e2ad52c\") " pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c885636-ca48-468f-b59b-a70323e7e02e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-run-multus-certs\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-run-k8s-cni-cncf-io\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c885636-ca48-468f-b59b-a70323e7e02e-cnibin\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3621c163-c9d0-4de5-9e47-e58b7af3729d-cni-binary-copy\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.240579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-cnibin\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-sysctl-conf\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-var-lib-kubelet\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e9179b5-f018-4459-bd59-5c2600534509-ovn-node-metrics-cert\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-var-lib-cni-multus\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-var-lib-kubelet\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-socket-dir-parent\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-run-netns\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.239953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c198208c-5766-492e-8ae7-40c2ea1a637c-host\") pod \"node-ca-4t8hn\" (UID: \"c198208c-5766-492e-8ae7-40c2ea1a637c\") " pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7a87c4-b91c-491e-9c5c-deab72585f4d-dbus\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-device-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-system-cni-dir\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-hostroot\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-cni-dir\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-daemon-config\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c885636-ca48-468f-b59b-a70323e7e02e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c885636-ca48-468f-b59b-a70323e7e02e-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.241167 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.240567    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:42.242033 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-socket-dir-parent\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.242033 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.240620    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs podName:67d8b168-67d4-4357-a091-52075e672f5f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:42.74060483 +0000 UTC m=+3.002281823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs") pod "network-metrics-daemon-knv9v" (UID: "67d8b168-67d4-4357-a091-52075e672f5f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:42.242033 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-multus-conf-dir\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.242033 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3621c163-c9d0-4de5-9e47-e58b7af3729d-host-run-netns\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.242033 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c198208c-5766-492e-8ae7-40c2ea1a637c-host\") pod \"node-ca-4t8hn\" (UID: \"c198208c-5766-492e-8ae7-40c2ea1a637c\") " pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.242033 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e740f22-4534-4351-8405-41938e2ad52c-tmp-dir\") pod \"node-resolver-x9lhv\" (UID: \"3e740f22-4534-4351-8405-41938e2ad52c\") " pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.242033 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c885636-ca48-468f-b59b-a70323e7e02e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.242033 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.240959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c885636-ca48-468f-b59b-a70323e7e02e-system-cni-dir\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.242033 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.241093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c198208c-5766-492e-8ae7-40c2ea1a637c-serviceca\") pod \"node-ca-4t8hn\" (UID: \"c198208c-5766-492e-8ae7-40c2ea1a637c\") " pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.245868 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.245806    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:43:42.247429 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.247405    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:42.247554 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.247436    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:42.247554 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.247451    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hh9dd for pod openshift-network-diagnostics/network-check-target-rr9t2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:42.247554 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.247517    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd podName:29869605-8fde-45a3-a354-424bafd17eb1 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:42.747500662 +0000 UTC m=+3.009177657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hh9dd" (UniqueName: "kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd") pod "network-check-target-rr9t2" (UID: "29869605-8fde-45a3-a354-424bafd17eb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:42.250200 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.250172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzxmg\" (UniqueName: \"kubernetes.io/projected/3e740f22-4534-4351-8405-41938e2ad52c-kube-api-access-vzxmg\") pod \"node-resolver-x9lhv\" (UID: \"3e740f22-4534-4351-8405-41938e2ad52c\") " pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.252560 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.252513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh7dj\" (UniqueName: \"kubernetes.io/projected/2c885636-ca48-468f-b59b-a70323e7e02e-kube-api-access-fh7dj\") pod \"multus-additional-cni-plugins-j8qqv\" (UID: \"2c885636-ca48-468f-b59b-a70323e7e02e\") " pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.252741 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.252722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvff\" (UniqueName: \"kubernetes.io/projected/c198208c-5766-492e-8ae7-40c2ea1a637c-kube-api-access-cpvff\") pod \"node-ca-4t8hn\" (UID: \"c198208c-5766-492e-8ae7-40c2ea1a637c\") " pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.252878 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.252859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flkmm\" (UniqueName: \"kubernetes.io/projected/3621c163-c9d0-4de5-9e47-e58b7af3729d-kube-api-access-flkmm\") pod \"multus-7nxf5\" (UID: \"3621c163-c9d0-4de5-9e47-e58b7af3729d\") " pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.253779 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.253703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkdn\" (UniqueName: \"kubernetes.io/projected/67d8b168-67d4-4357-a091-52075e672f5f-kube-api-access-5tkdn\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:42.275106 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.275058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal" event={"ID":"94f8490c5f2548f89f7eaeda969f1633","Type":"ContainerStarted","Data":"0fc569f104182f8b90d75b8c896c0fa324cf99d71e8f249d93523548aa231cec"}
Sep 16 18:43:42.276974 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.276943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal" event={"ID":"c92465bf8a81cf25e8e179fc079decff","Type":"ContainerStarted","Data":"f6d03c8a37608215265ac4449356f2e18ce849c7dbf77a620035c7196d0d7e64"}
Sep 16 18:43:42.340914 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.340820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-systemd\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.340914 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.340856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z8np\" (UniqueName: \"kubernetes.io/projected/8e83e603-da27-4035-888a-811cc80ca89f-kube-api-access-8z8np\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.340914 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.340874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e9179b5-f018-4459-bd59-5c2600534509-env-overrides\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.341141 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.340941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.341141 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.341018    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:42.341141 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.341078    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret podName:ed7a87c4-b91c-491e-9c5c-deab72585f4d nodeName:}" failed. No retries permitted until 2026-09-16 18:43:42.84105748 +0000 UTC m=+3.102734478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret") pod "global-pull-secret-syncer-4cwkd" (UID: "ed7a87c4-b91c-491e-9c5c-deab72585f4d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:42.341282 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-systemd\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341282 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-registration-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.341371 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-systemd-units\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.341371 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bff8a-35f8-4491-a764-3e1318d4b95f-konnectivity-ca\") pod \"konnectivity-agent-wwmwd\" (UID: \"cb9bff8a-35f8-4491-a764-3e1318d4b95f\") " pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:43:42.341371 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-sys\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-socket-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.341479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-sys-fs\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.341479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-kubernetes\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-host\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e83e603-da27-4035-888a-811cc80ca89f-etc-tuned\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-run-openvswitch\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.341665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e9179b5-f018-4459-bd59-5c2600534509-ovnkube-script-lib\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.341665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltm2m\" (UniqueName: \"kubernetes.io/projected/9e9179b5-f018-4459-bd59-5c2600534509-kube-api-access-ltm2m\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.341665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.341665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341576    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e9179b5-f018-4459-bd59-5c2600534509-env-overrides\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.341665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb9bff8a-35f8-4491-a764-3e1318d4b95f-agent-certs\") pod \"konnectivity-agent-wwmwd\" (UID: \"cb9bff8a-35f8-4491-a764-3e1318d4b95f\") " pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:43:42.341665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-sysctl-conf\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-var-lib-kubelet\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e9179b5-f018-4459-bd59-5c2600534509-ovn-node-metrics-cert\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7a87c4-b91c-491e-9c5c-deab72585f4d-dbus\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-device-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-kubelet\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7a87c4-b91c-491e-9c5c-deab72585f4d-kubelet-config\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-lib-modules\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e83e603-da27-4035-888a-811cc80ca89f-tmp\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-sysctl-d\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.341941 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-slash\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-log-socket\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.341994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-cni-bin\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfd5\" (UniqueName: \"kubernetes.io/projected/db3f5719-c1f7-40ea-83fb-395aaca843e4-kube-api-access-zcfd5\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-run-netns\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-run-systemd\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-etc-openvswitch\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-node-log\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbe1e74d-fc0c-48b5-aedc-a3822f634fa5-iptables-alerter-script\") pod \"iptables-alerter-bfg62\" (UID: \"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5\") " pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbkq\" (UniqueName: \"kubernetes.io/projected/dbe1e74d-fc0c-48b5-aedc-a3822f634fa5-kube-api-access-rhbkq\") pod \"iptables-alerter-bfg62\" (UID: \"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5\") " pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-run-ovn-kubernetes\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-run\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.342308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-run-ovn\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-cni-netd\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-modprobe-d\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-systemd-units\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-slash\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-device-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-var-lib-kubelet\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-sysconfig\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-sys-fs\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbe1e74d-fc0c-48b5-aedc-a3822f634fa5-host-slash\") pod \"iptables-alerter-bfg62\" (UID: \"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5\") " pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-kubernetes\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-var-lib-openvswitch\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-modprobe-d\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e9179b5-f018-4459-bd59-5c2600534509-ovnkube-config\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-host\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.342866 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-registration-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-run-openvswitch\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-sysctl-conf\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-sysconfig\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-socket-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-sys\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.342987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-etc-sysctl-d\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bff8a-35f8-4491-a764-3e1318d4b95f-konnectivity-ca\") pod \"konnectivity-agent-wwmwd\" (UID: \"cb9bff8a-35f8-4491-a764-3e1318d4b95f\") " pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7a87c4-b91c-491e-9c5c-deab72585f4d-dbus\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-log-socket\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-node-log\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-cni-bin\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-etc-openvswitch\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e9179b5-f018-4459-bd59-5c2600534509-ovnkube-script-lib\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-var-lib-openvswitch\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7a87c4-b91c-491e-9c5c-deab72585f4d-kubelet-config\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-kubelet\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e9179b5-f018-4459-bd59-5c2600534509-ovnkube-config\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.343770 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-run-systemd\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.344601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-lib-modules\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.344601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbe1e74d-fc0c-48b5-aedc-a3822f634fa5-host-slash\") pod \"iptables-alerter-bfg62\" (UID: \"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5\") " pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.344601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e83e603-da27-4035-888a-811cc80ca89f-run\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.344601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db3f5719-c1f7-40ea-83fb-395aaca843e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.344601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-run-ovn-kubernetes\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.344601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-run-ovn\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.344601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-cni-netd\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.344601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbe1e74d-fc0c-48b5-aedc-a3822f634fa5-iptables-alerter-script\") pod \"iptables-alerter-bfg62\" (UID: \"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5\") " pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.344601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.343880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e9179b5-f018-4459-bd59-5c2600534509-host-run-netns\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.345846 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.345498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb9bff8a-35f8-4491-a764-3e1318d4b95f-agent-certs\") pod \"konnectivity-agent-wwmwd\" (UID: \"cb9bff8a-35f8-4491-a764-3e1318d4b95f\") " pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:43:42.345846 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.345646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e9179b5-f018-4459-bd59-5c2600534509-ovn-node-metrics-cert\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.346129 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.346110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e83e603-da27-4035-888a-811cc80ca89f-etc-tuned\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.349877 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.346830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e83e603-da27-4035-888a-811cc80ca89f-tmp\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.355099 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.355074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltm2m\" (UniqueName: \"kubernetes.io/projected/9e9179b5-f018-4459-bd59-5c2600534509-kube-api-access-ltm2m\") pod \"ovnkube-node-6dwh7\" (UID: \"9e9179b5-f018-4459-bd59-5c2600534509\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.355331 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.355250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z8np\" (UniqueName: \"kubernetes.io/projected/8e83e603-da27-4035-888a-811cc80ca89f-kube-api-access-8z8np\") pod \"tuned-tsbtz\" (UID: \"8e83e603-da27-4035-888a-811cc80ca89f\") " pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.356926 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.356836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbkq\" (UniqueName: \"kubernetes.io/projected/dbe1e74d-fc0c-48b5-aedc-a3822f634fa5-kube-api-access-rhbkq\") pod \"iptables-alerter-bfg62\" (UID: \"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5\") " pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.357510 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.357443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfd5\" (UniqueName: \"kubernetes.io/projected/db3f5719-c1f7-40ea-83fb-395aaca843e4-kube-api-access-zcfd5\") pod \"aws-ebs-csi-driver-node-nwdht\" (UID: \"db3f5719-c1f7-40ea-83fb-395aaca843e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.432267 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.432226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x9lhv"
Sep 16 18:43:42.441293 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.441270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4t8hn"
Sep 16 18:43:42.454619 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.454590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7nxf5"
Sep 16 18:43:42.460570 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.460550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j8qqv"
Sep 16 18:43:42.470640 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.470619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:43:42.480577 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.480552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bfg62"
Sep 16 18:43:42.488467 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.488445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht"
Sep 16 18:43:42.500306 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.500288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tsbtz"
Sep 16 18:43:42.506140 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.506120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:43:42.746262 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.746160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:42.746430 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.746362    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:42.746507 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.746442    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs podName:67d8b168-67d4-4357-a091-52075e672f5f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:43.746421041 +0000 UTC m=+4.008098043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs") pod "network-metrics-daemon-knv9v" (UID: "67d8b168-67d4-4357-a091-52075e672f5f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:42.847075 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.847039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:42.847278 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:42.847097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:42.847278 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.847237    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:42.847391 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.847303    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret podName:ed7a87c4-b91c-491e-9c5c-deab72585f4d nodeName:}" failed. No retries permitted until 2026-09-16 18:43:43.847282256 +0000 UTC m=+4.108959249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret") pod "global-pull-secret-syncer-4cwkd" (UID: "ed7a87c4-b91c-491e-9c5c-deab72585f4d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:42.847647 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.847615    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:42.847647 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.847648    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:42.847808 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.847663    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hh9dd for pod openshift-network-diagnostics/network-check-target-rr9t2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:42.847808 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:42.847736    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd podName:29869605-8fde-45a3-a354-424bafd17eb1 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:43.847706947 +0000 UTC m=+4.109383943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hh9dd" (UniqueName: "kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd") pod "network-check-target-rr9t2" (UID: "29869605-8fde-45a3-a354-424bafd17eb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:43.176863 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:43.176731    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:42 +0000 UTC" deadline="2028-05-03 23:05:00.493418392 +0000 UTC"
Sep 16 18:43:43.176863 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:43.176815    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14284h21m17.316607337s"
Sep 16 18:43:43.269886 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:43.269855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:43.270051 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:43.269992    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:43:43.436685 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:43.436609    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:43.451041 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:43.451012    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e83e603_da27_4035_888a_811cc80ca89f.slice/crio-7dde3b67cbee0fb444905a41d88ed93b157383c8f3af0a1f99cfc48460f02718 WatchSource:0}: Error finding container 7dde3b67cbee0fb444905a41d88ed93b157383c8f3af0a1f99cfc48460f02718: Status 404 returned error can't find the container with id 7dde3b67cbee0fb444905a41d88ed93b157383c8f3af0a1f99cfc48460f02718
Sep 16 18:43:43.472218 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:43:43.472193    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c885636_ca48_468f_b59b_a70323e7e02e.slice/crio-7ab2168d3caf4b449ca355fa0fcb6cce464f72a02757b42eb371fc941da6ce2a WatchSource:0}: Error finding container 7ab2168d3caf4b449ca355fa0fcb6cce464f72a02757b42eb371fc941da6ce2a: Status 404 returned error can't find the container with id 7ab2168d3caf4b449ca355fa0fcb6cce464f72a02757b42eb371fc941da6ce2a
Sep 16 18:43:43.754517 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:43.754289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:43.754625 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:43.754443    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:43.754668 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:43.754631    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs podName:67d8b168-67d4-4357-a091-52075e672f5f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:45.754614992 +0000 UTC m=+6.016291988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs") pod "network-metrics-daemon-knv9v" (UID: "67d8b168-67d4-4357-a091-52075e672f5f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:43.855584 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:43.855536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:43.855761 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:43.855600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:43.855761 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:43.855746    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:43.855866 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:43.855762    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:43.855866 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:43.855789    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:43.855866 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:43.855801    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hh9dd for pod openshift-network-diagnostics/network-check-target-rr9t2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:43.855866 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:43.855821    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret podName:ed7a87c4-b91c-491e-9c5c-deab72585f4d nodeName:}" failed. No retries permitted until 2026-09-16 18:43:45.85580006 +0000 UTC m=+6.117477055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret") pod "global-pull-secret-syncer-4cwkd" (UID: "ed7a87c4-b91c-491e-9c5c-deab72585f4d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:43.855866 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:43.855854    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd podName:29869605-8fde-45a3-a354-424bafd17eb1 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:45.855835836 +0000 UTC m=+6.117512828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hh9dd" (UniqueName: "kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd") pod "network-check-target-rr9t2" (UID: "29869605-8fde-45a3-a354-424bafd17eb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:44.180912 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.180728    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:42 +0000 UTC" deadline="2028-04-09 15:20:51.048491352 +0000 UTC"
Sep 16 18:43:44.180912 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.180775    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13700h37m6.867721982s"
Sep 16 18:43:44.273833 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.273043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:44.273833 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:44.273194    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:43:44.273833 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.273652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:44.273833 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:44.273771    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:43:44.296029 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.295289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal" event={"ID":"c92465bf8a81cf25e8e179fc079decff","Type":"ContainerStarted","Data":"3757533eb78d01d782975fcd7dba6ab2cffdf69721722852adb338807bd153be"}
Sep 16 18:43:44.300312 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.300235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4t8hn" event={"ID":"c198208c-5766-492e-8ae7-40c2ea1a637c","Type":"ContainerStarted","Data":"dc797ce808b2cb0b6c83548d1a0e7679ded162033a1d5a94913d0162e4df9dee"}
Sep 16 18:43:44.312916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.312881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bfg62" event={"ID":"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5","Type":"ContainerStarted","Data":"0b535b4bab733cde4d0b20d696756ee9c4b9b050fb0b852d383b8edae9009cbb"}
Sep 16 18:43:44.322552 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.322469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht" event={"ID":"db3f5719-c1f7-40ea-83fb-395aaca843e4","Type":"ContainerStarted","Data":"e4fbae5a8de70aa9bfc87663e188fb02cb47b1b9b462f541984d83ed1440b7da"}
Sep 16 18:43:44.332723 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.332690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x9lhv" event={"ID":"3e740f22-4534-4351-8405-41938e2ad52c","Type":"ContainerStarted","Data":"9a071dcc43556171bc0916b27e4d8d726618910997db4bde4ab46a21b99af1fb"}
Sep 16 18:43:44.343311 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.343253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7nxf5" event={"ID":"3621c163-c9d0-4de5-9e47-e58b7af3729d","Type":"ContainerStarted","Data":"7c7ed910c2ab0cf340db1760e515614ffafe9ae8597e043d80653d65343cb49a"}
Sep 16 18:43:44.351752 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.351681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tsbtz" event={"ID":"8e83e603-da27-4035-888a-811cc80ca89f","Type":"ContainerStarted","Data":"7dde3b67cbee0fb444905a41d88ed93b157383c8f3af0a1f99cfc48460f02718"}
Sep 16 18:43:44.356466 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.356434    2580 generic.go:356] "Generic (PLEG): container finished" podID="94f8490c5f2548f89f7eaeda969f1633" containerID="2c24ca928a694115d4f0684de6174fb8737447f5c626639aebefd85901dc6609" exitCode=0
Sep 16 18:43:44.356591 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.356525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal" event={"ID":"94f8490c5f2548f89f7eaeda969f1633","Type":"ContainerDied","Data":"2c24ca928a694115d4f0684de6174fb8737447f5c626639aebefd85901dc6609"}
Sep 16 18:43:44.365479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.365447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8qqv" event={"ID":"2c885636-ca48-468f-b59b-a70323e7e02e","Type":"ContainerStarted","Data":"7ab2168d3caf4b449ca355fa0fcb6cce464f72a02757b42eb371fc941da6ce2a"}
Sep 16 18:43:44.373259 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.372017    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-95.ec2.internal" podStartSLOduration=2.371996971 podStartE2EDuration="2.371996971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:44.315151708 +0000 UTC m=+4.576828724" watchObservedRunningTime="2026-09-16 18:43:44.371996971 +0000 UTC m=+4.633673987"
Sep 16 18:43:44.380480 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.380407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wwmwd" event={"ID":"cb9bff8a-35f8-4491-a764-3e1318d4b95f","Type":"ContainerStarted","Data":"87714df84c42ce1983a9e85c0b2b4c1045cd6e75309630a6bb55d5a0aa95a2d1"}
Sep 16 18:43:44.399714 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:44.399360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" event={"ID":"9e9179b5-f018-4459-bd59-5c2600534509","Type":"ContainerStarted","Data":"7536a5de8de5accfe3017458b34b63a9ecd15a9e691b5f4a0943dfcca8073f89"}
Sep 16 18:43:45.272101 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:45.271158    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:45.272101 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:45.271287    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:43:45.431732 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:45.431678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal" event={"ID":"94f8490c5f2548f89f7eaeda969f1633","Type":"ContainerStarted","Data":"0029d7372041cbe191b368a42e66d66d8d2c2b588aee9e2b1ac928babd419e6d"}
Sep 16 18:43:45.773990 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:45.773568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:45.773990 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:45.773778    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:45.773990 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:45.773848    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs podName:67d8b168-67d4-4357-a091-52075e672f5f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:49.773827872 +0000 UTC m=+10.035504867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs") pod "network-metrics-daemon-knv9v" (UID: "67d8b168-67d4-4357-a091-52075e672f5f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:45.874185 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:45.874140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:45.874371 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:45.874217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:45.874585 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:45.874423    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:45.874585 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:45.874499    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret podName:ed7a87c4-b91c-491e-9c5c-deab72585f4d nodeName:}" failed. No retries permitted until 2026-09-16 18:43:49.874477852 +0000 UTC m=+10.136154846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret") pod "global-pull-secret-syncer-4cwkd" (UID: "ed7a87c4-b91c-491e-9c5c-deab72585f4d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:45.875038 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:45.875010    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:45.875038 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:45.875034    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:45.875200 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:45.875049    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hh9dd for pod openshift-network-diagnostics/network-check-target-rr9t2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:45.875200 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:45.875100    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd podName:29869605-8fde-45a3-a354-424bafd17eb1 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:49.875083748 +0000 UTC m=+10.136760741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hh9dd" (UniqueName: "kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd") pod "network-check-target-rr9t2" (UID: "29869605-8fde-45a3-a354-424bafd17eb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:46.272372 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:46.272274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:46.272760 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:46.272428    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:43:46.272760 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:46.272516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:46.272760 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:46.272597    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:43:47.269819 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:47.269785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:47.270023 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:47.269932    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:43:48.272864 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:48.272224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:48.272864 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:48.272387    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:43:48.272864 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:48.272705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:48.272864 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:48.272813    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:43:49.270585 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:49.270549    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:49.270791 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:49.270669    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:43:49.811709 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:49.811673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:49.812192 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:49.811840    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:49.812192 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:49.811922    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs podName:67d8b168-67d4-4357-a091-52075e672f5f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:57.811885934 +0000 UTC m=+18.073562930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs") pod "network-metrics-daemon-knv9v" (UID: "67d8b168-67d4-4357-a091-52075e672f5f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:49.912215 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:49.912175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:49.912420 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:49.912234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:49.912420 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:49.912371    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:49.912420 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:49.912417    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret podName:ed7a87c4-b91c-491e-9c5c-deab72585f4d nodeName:}" failed. No retries permitted until 2026-09-16 18:43:57.912404126 +0000 UTC m=+18.174081115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret") pod "global-pull-secret-syncer-4cwkd" (UID: "ed7a87c4-b91c-491e-9c5c-deab72585f4d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:49.912859 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:49.912840    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:49.912952 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:49.912864    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:49.912952 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:49.912874    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hh9dd for pod openshift-network-diagnostics/network-check-target-rr9t2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:49.912952 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:49.912930    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd podName:29869605-8fde-45a3-a354-424bafd17eb1 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:57.912915685 +0000 UTC m=+18.174592697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hh9dd" (UniqueName: "kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd") pod "network-check-target-rr9t2" (UID: "29869605-8fde-45a3-a354-424bafd17eb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:50.271342 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:50.271259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:50.271493 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:50.271265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:50.271493 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:50.271382    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:43:50.271576 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:50.271483    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:43:51.269724 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:51.269682    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:51.270358 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:51.269834    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:43:52.270337 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:52.270254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:52.270915 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:52.270415    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:43:52.270915 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:52.270799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:52.271041 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:52.270910    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:43:53.270551 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:53.270511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:53.270975 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:53.270650    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:43:54.269768 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:54.269735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:54.269955 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:54.269879    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:43:54.269955 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:54.269932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:54.270083 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:54.270051    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:43:55.270430 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:55.270391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:55.270968 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:55.270516    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:43:56.269859 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:56.269821    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:56.270048 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:56.269873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:56.270048 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:56.269994    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:43:56.270165 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:56.270139    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:43:57.270012 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:57.269978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:57.270502 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:57.270111    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:43:57.871844 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:57.871809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:57.872056 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:57.871987    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:57.872124 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:57.872059    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs podName:67d8b168-67d4-4357-a091-52075e672f5f nodeName:}" failed. No retries permitted until 2026-09-16 18:44:13.872038875 +0000 UTC m=+34.133715868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs") pod "network-metrics-daemon-knv9v" (UID: "67d8b168-67d4-4357-a091-52075e672f5f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:57.972309 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:57.972276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:57.972465 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:57.972321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:57.972465 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:57.972418    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:57.972465 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:57.972438    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:57.972465 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:57.972458    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:57.972465 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:57.972466    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret podName:ed7a87c4-b91c-491e-9c5c-deab72585f4d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:13.972452545 +0000 UTC m=+34.234129535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret") pod "global-pull-secret-syncer-4cwkd" (UID: "ed7a87c4-b91c-491e-9c5c-deab72585f4d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:57.972615 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:57.972472    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hh9dd for pod openshift-network-diagnostics/network-check-target-rr9t2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:57.972615 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:57.972514    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd podName:29869605-8fde-45a3-a354-424bafd17eb1 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:13.97250335 +0000 UTC m=+34.234180340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hh9dd" (UniqueName: "kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd") pod "network-check-target-rr9t2" (UID: "29869605-8fde-45a3-a354-424bafd17eb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:58.270149 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:58.270067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:43:58.270560 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:58.270072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:43:58.270560 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:58.270224    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:43:58.270560 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:58.270267    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:43:59.270577 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:43:59.270541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:43:59.271126 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:43:59.270664    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:00.270803 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:00.270768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:00.271271 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:00.270876    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:00.271271 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:00.270964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:00.271271 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:00.271074    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:01.270492 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:01.270454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:01.270656 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:01.270563    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:02.270341 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.269954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:02.270708 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.269979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:02.270708 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:02.270467    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:02.270708 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:02.270576    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:02.473435 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.473406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wwmwd" event={"ID":"cb9bff8a-35f8-4491-a764-3e1318d4b95f","Type":"ContainerStarted","Data":"a6fde2756e61b18b5ae1f19c2fa90d57ad9e8d34429afaad0a36c06936c1a233"}
Sep 16 18:44:02.478258 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.478220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" event={"ID":"9e9179b5-f018-4459-bd59-5c2600534509","Type":"ContainerStarted","Data":"3a2a1297a0a1762ec469eea66685e9bec174f6ccf9c5cf0787684a5c1b29120f"}
Sep 16 18:44:02.478361 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.478321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" event={"ID":"9e9179b5-f018-4459-bd59-5c2600534509","Type":"ContainerStarted","Data":"985405d9872027d1ce53664f82d2eb71b45945b6b9551316ec5b5e5c656bf8ef"}
Sep 16 18:44:02.478426 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.478370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" event={"ID":"9e9179b5-f018-4459-bd59-5c2600534509","Type":"ContainerStarted","Data":"edd8e9a86bfe57103d47acd644ac28a2d0eb3cd11e31349d38ee72e189a1308e"}
Sep 16 18:44:02.478426 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.478418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" event={"ID":"9e9179b5-f018-4459-bd59-5c2600534509","Type":"ContainerStarted","Data":"2121584026fc1ac9ea3c77af078d4b9b7e5eb89a019a19bf9d4ba414967b0de3"}
Sep 16 18:44:02.480494 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.480467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4t8hn" event={"ID":"c198208c-5766-492e-8ae7-40c2ea1a637c","Type":"ContainerStarted","Data":"7e7d2f6164b517374d93ddd5208c58e8526dc6487f70f8b78d7419a07a667736"}
Sep 16 18:44:02.482756 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.482732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht" event={"ID":"db3f5719-c1f7-40ea-83fb-395aaca843e4","Type":"ContainerStarted","Data":"31a8231ed6a323db0213a5eca7c617564360543e285e20562ab64590b61e9ee8"}
Sep 16 18:44:02.484479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.484451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x9lhv" event={"ID":"3e740f22-4534-4351-8405-41938e2ad52c","Type":"ContainerStarted","Data":"ce9ad515ac00800313a3aa6b95f4676b8f5205dd9fa527d14ece2ca01609a83b"}
Sep 16 18:44:02.486450 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.486418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7nxf5" event={"ID":"3621c163-c9d0-4de5-9e47-e58b7af3729d","Type":"ContainerStarted","Data":"cbca9d5ba5392bf3c4b33a79660a520f8c5b95098bbf4bda113f119d77dbc540"}
Sep 16 18:44:02.489405 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.489340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tsbtz" event={"ID":"8e83e603-da27-4035-888a-811cc80ca89f","Type":"ContainerStarted","Data":"cc276b385c36119389e943e115a975b1261e3f0da81dca786d51f59a5f6cb6d3"}
Sep 16 18:44:02.491465 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.491445    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c885636-ca48-468f-b59b-a70323e7e02e" containerID="4bd39f93ebcb407cdc4b4d4282bc2ebef7b468d57cffedcdbc7b9ee0c92e819d" exitCode=0
Sep 16 18:44:02.491551 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.491475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8qqv" event={"ID":"2c885636-ca48-468f-b59b-a70323e7e02e","Type":"ContainerDied","Data":"4bd39f93ebcb407cdc4b4d4282bc2ebef7b468d57cffedcdbc7b9ee0c92e819d"}
Sep 16 18:44:02.493392 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.493339    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-95.ec2.internal" podStartSLOduration=20.493322712 podStartE2EDuration="20.493322712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:45.448946705 +0000 UTC m=+5.710623724" watchObservedRunningTime="2026-09-16 18:44:02.493322712 +0000 UTC m=+22.754999727"
Sep 16 18:44:02.494410 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.494369    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wwmwd" podStartSLOduration=4.06545127 podStartE2EDuration="22.494353982s" totalImagesPullingTime="18.428902712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:02.492841506 +0000 UTC m=+22.754518519" watchObservedRunningTime="2026-09-16 18:44:02.494353982 +0000 UTC m=+22.756030994"
Sep 16 18:44:02.528122 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.528069    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x9lhv" podStartSLOduration=4.089738973 podStartE2EDuration="22.528048007s" totalImagesPullingTime="18.438309034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:02.527200824 +0000 UTC m=+22.788877836" watchObservedRunningTime="2026-09-16 18:44:02.528048007 +0000 UTC m=+22.789725019"
Sep 16 18:44:02.541808 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.541771    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4t8hn" podStartSLOduration=4.115704501 podStartE2EDuration="22.541754867s" totalImagesPullingTime="18.426050366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:02.541570623 +0000 UTC m=+22.803247637" watchObservedRunningTime="2026-09-16 18:44:02.541754867 +0000 UTC m=+22.803431879"
Sep 16 18:44:02.562397 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.562347    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7nxf5" podStartSLOduration=4.04219309 podStartE2EDuration="22.562327457s" totalImagesPullingTime="18.520134367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:02.5621734 +0000 UTC m=+22.823850412" watchObservedRunningTime="2026-09-16 18:44:02.562327457 +0000 UTC m=+22.824004471"
Sep 16 18:44:02.581242 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:02.581187    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tsbtz" podStartSLOduration=4.114540313 podStartE2EDuration="22.581168936s" totalImagesPullingTime="18.466628623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:02.58073074 +0000 UTC m=+22.842407754" watchObservedRunningTime="2026-09-16 18:44:02.581168936 +0000 UTC m=+22.842845947"
Sep 16 18:44:03.122172 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:03.122144    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:44:03.122665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:03.122650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:44:03.230520 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:03.230485    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:44:03.269663 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:03.269635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:03.269789 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:03.269737    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:03.498082 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:03.498005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" event={"ID":"9e9179b5-f018-4459-bd59-5c2600534509","Type":"ContainerStarted","Data":"93b39cb52a7aba07be876e9f751074999d0c9ea02021057d96a698e7d7b262bd"}
Sep 16 18:44:03.498082 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:03.498082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" event={"ID":"9e9179b5-f018-4459-bd59-5c2600534509","Type":"ContainerStarted","Data":"19e344eb300bc5bb1087da015f9664967c27559887ab43801cf6f09668bdcf48"}
Sep 16 18:44:03.500071 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:03.500035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht" event={"ID":"db3f5719-c1f7-40ea-83fb-395aaca843e4","Type":"ContainerStarted","Data":"69fef1831d2e02cdf7d7a38e8281638baaa7c3b0265eab912bbac54cd21dd8fd"}
Sep 16 18:44:03.500931 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:03.500909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:44:03.501272 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:03.501253    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wwmwd"
Sep 16 18:44:04.194401 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:04.194306    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:44:03.230516798Z","UUID":"ba129303-1bb2-411f-a297-878843eee417","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:44:04.196808 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:04.196768    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:44:04.196954 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:04.196839    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:44:04.269938 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:04.269911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:04.270215 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:04.270040    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:04.270488 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:04.270330    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:04.270488 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:04.270443    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:04.504741 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:04.504704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bfg62" event={"ID":"dbe1e74d-fc0c-48b5-aedc-a3822f634fa5","Type":"ContainerStarted","Data":"ee8e04b5abdbedb649d9ea9cfb1f8dea73058d88282198405d1e323956a32b09"}
Sep 16 18:44:04.507236 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:04.507209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht" event={"ID":"db3f5719-c1f7-40ea-83fb-395aaca843e4","Type":"ContainerStarted","Data":"8513fb5520d36fdf1f1b4eb3a6252be47b077854a2521f7fa2841e4ba88cc67c"}
Sep 16 18:44:04.518294 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:04.518244    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bfg62" podStartSLOduration=6.092611882 podStartE2EDuration="24.518224315s" totalImagesPullingTime="18.425612433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:04.51741943 +0000 UTC m=+24.779096673" watchObservedRunningTime="2026-09-16 18:44:04.518224315 +0000 UTC m=+24.779901329"
Sep 16 18:44:04.532704 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:04.532656    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwdht" podStartSLOduration=4.016921605 podStartE2EDuration="24.532636296s" totalImagesPullingTime="20.515714691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:04.532251047 +0000 UTC m=+24.793928059" watchObservedRunningTime="2026-09-16 18:44:04.532636296 +0000 UTC m=+24.794313309"
Sep 16 18:44:05.269567 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:05.269531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:05.269740 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:05.269654    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:05.512751 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:05.512703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" event={"ID":"9e9179b5-f018-4459-bd59-5c2600534509","Type":"ContainerStarted","Data":"df530cf8fbd3713ce8493f73a46f9547e260e5c5c2235957ed658e1fc56dd3d6"}
Sep 16 18:44:06.270228 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:06.270189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:06.270391 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:06.270333    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:06.270821 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:06.270796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:06.270931 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:06.270915    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:07.270484 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:07.270290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:07.270932 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:07.270559    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:07.519390 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:07.519312    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c885636-ca48-468f-b59b-a70323e7e02e" containerID="ec007950daba25ca69385180a4ea7fc3ecaf9429e48fab69c22eb88080afa2f6" exitCode=0
Sep 16 18:44:07.519675 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:07.519401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8qqv" event={"ID":"2c885636-ca48-468f-b59b-a70323e7e02e","Type":"ContainerDied","Data":"ec007950daba25ca69385180a4ea7fc3ecaf9429e48fab69c22eb88080afa2f6"}
Sep 16 18:44:07.523452 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:07.523428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" event={"ID":"9e9179b5-f018-4459-bd59-5c2600534509","Type":"ContainerStarted","Data":"6b343150d2ad8cfceb32527223c31741d4448852973792777ff67d013f09946a"}
Sep 16 18:44:07.523745 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:07.523722    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:44:07.523822 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:07.523796    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:44:07.540001 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:07.539975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:44:07.562186 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:07.562129    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7" podStartSLOduration=9.067109503 podStartE2EDuration="27.562114586s" totalImagesPullingTime="18.495005083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.561264104 +0000 UTC m=+27.822941126" watchObservedRunningTime="2026-09-16 18:44:07.562114586 +0000 UTC m=+27.823791597"
Sep 16 18:44:08.270324 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.270080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:08.270429 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.270115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:08.270485 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:08.270459    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:08.270550 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:08.270494    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:08.531364 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.531278    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c885636-ca48-468f-b59b-a70323e7e02e" containerID="bc60d2b5d3f43f6dbf5f3597e807e3fa6e93f5a4a25cb74e002b00f9453905a0" exitCode=0
Sep 16 18:44:08.531500 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.531358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8qqv" event={"ID":"2c885636-ca48-468f-b59b-a70323e7e02e","Type":"ContainerDied","Data":"bc60d2b5d3f43f6dbf5f3597e807e3fa6e93f5a4a25cb74e002b00f9453905a0"}
Sep 16 18:44:08.531839 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.531823    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:44:08.547652 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.547625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:44:08.856794 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.856707    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4cwkd"]
Sep 16 18:44:08.856963 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.856823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:08.856963 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:08.856941    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:08.860476 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.860207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rr9t2"]
Sep 16 18:44:08.860476 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.860337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:08.860476 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:08.860434    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:08.860963 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.860928    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-knv9v"]
Sep 16 18:44:08.861061 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:08.861034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:08.861127 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:08.861103    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:09.454055 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:09.453983    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x9lhv_3e740f22-4534-4351-8405-41938e2ad52c/dns-node-resolver/0.log"
Sep 16 18:44:09.536654 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:09.536620    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c885636-ca48-468f-b59b-a70323e7e02e" containerID="5de3a377e8050e7d85d839f4e29c0a5eaedaf14672c6c0fa87ae53f2dae1ec61" exitCode=0
Sep 16 18:44:09.536828 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:09.536706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8qqv" event={"ID":"2c885636-ca48-468f-b59b-a70323e7e02e","Type":"ContainerDied","Data":"5de3a377e8050e7d85d839f4e29c0a5eaedaf14672c6c0fa87ae53f2dae1ec61"}
Sep 16 18:44:10.037640 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:10.037614    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4t8hn_c198208c-5766-492e-8ae7-40c2ea1a637c/node-ca/0.log"
Sep 16 18:44:10.271695 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:10.271657    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:10.271848 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:10.271739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:10.271848 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:10.271784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:10.271848 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:10.271800    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:10.271988 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:10.271866    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:10.272036 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:10.272003    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:12.270076 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:12.269839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:12.270624 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:12.269856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:12.270624 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:12.270179    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:12.270624 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:12.269881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:12.270624 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:12.270287    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:12.270624 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:12.270321    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:13.888373 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:13.888344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:13.888797 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:13.888482    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:13.888797 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:13.888536    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs podName:67d8b168-67d4-4357-a091-52075e672f5f nodeName:}" failed. No retries permitted until 2026-09-16 18:44:45.888522031 +0000 UTC m=+66.150199021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs") pod "network-metrics-daemon-knv9v" (UID: "67d8b168-67d4-4357-a091-52075e672f5f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:13.989445 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:13.989420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:13.989560 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:13.989458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:13.989612 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:13.989565    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:13.989612 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:13.989583    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:13.989612 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:13.989604    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:13.989718 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:13.989615    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hh9dd for pod openshift-network-diagnostics/network-check-target-rr9t2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:13.989718 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:13.989623    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret podName:ed7a87c4-b91c-491e-9c5c-deab72585f4d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:45.989608341 +0000 UTC m=+66.251285354 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret") pod "global-pull-secret-syncer-4cwkd" (UID: "ed7a87c4-b91c-491e-9c5c-deab72585f4d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:13.989718 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:13.989666    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd podName:29869605-8fde-45a3-a354-424bafd17eb1 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:45.989655069 +0000 UTC m=+66.251332059 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hh9dd" (UniqueName: "kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd") pod "network-check-target-rr9t2" (UID: "29869605-8fde-45a3-a354-424bafd17eb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:14.270381 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:14.270299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:14.270571 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:14.270312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:14.270571 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:14.270401    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:14.270571 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:14.270479    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:14.270571 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:14.270520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:14.270571 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:14.270570    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:14.552611 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:14.552524    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c885636-ca48-468f-b59b-a70323e7e02e" containerID="1340ac1fe95b5afaac2215054c9bb05cdefe51f8d5f4e8afc208aa6649d7a37e" exitCode=0
Sep 16 18:44:14.552611 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:14.552581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8qqv" event={"ID":"2c885636-ca48-468f-b59b-a70323e7e02e","Type":"ContainerDied","Data":"1340ac1fe95b5afaac2215054c9bb05cdefe51f8d5f4e8afc208aa6649d7a37e"}
Sep 16 18:44:15.557474 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:15.557444    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c885636-ca48-468f-b59b-a70323e7e02e" containerID="22f42c4865f900e9ced8524d2002b4d6a6079e65f80cee075bf054056a7a9d14" exitCode=0
Sep 16 18:44:15.557849 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:15.557500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8qqv" event={"ID":"2c885636-ca48-468f-b59b-a70323e7e02e","Type":"ContainerDied","Data":"22f42c4865f900e9ced8524d2002b4d6a6079e65f80cee075bf054056a7a9d14"}
Sep 16 18:44:16.269677 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:16.269639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:16.269867 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:16.269639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:16.269867 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:16.269734    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:16.269867 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:16.269809    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:16.269867 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:16.269856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:16.270034 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:16.269971    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:16.563205 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:16.563136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8qqv" event={"ID":"2c885636-ca48-468f-b59b-a70323e7e02e","Type":"ContainerStarted","Data":"8a5435834f8dd15ab292f7a460788fd19f35b466b85a25563482c3d647b66786"}
Sep 16 18:44:16.583948 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:16.583878    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j8qqv" podStartSLOduration=8.26934723 podStartE2EDuration="36.583861696s" totalImagesPullingTime="28.314514466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:16.583202243 +0000 UTC m=+36.844879260" watchObservedRunningTime="2026-09-16 18:44:16.583861696 +0000 UTC m=+36.845538709"
Sep 16 18:44:18.270592 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:18.270561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:18.271077 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:18.270569    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:18.271077 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:18.270665    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:18.271077 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:18.270760    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:18.271077 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:18.270777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:18.271077 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:18.270844    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:20.270931 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:20.270881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:20.271355 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:20.270995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:20.271355 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:20.270997    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:20.271355 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:20.271069    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:20.271355 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:20.271087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:20.271355 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:20.271228    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:22.270079 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:22.270047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:22.270535 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:22.270094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:22.270535 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:22.270047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:22.270535 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:22.270175    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:22.270535 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:22.270252    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:22.270535 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:22.270310    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:24.270460 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:24.270424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:24.270846 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:24.270424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:24.270846 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:24.270541    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:24.270846 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:24.270436    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:24.270846 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:24.270611    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:24.270846 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:24.270723    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:26.269914 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:26.269871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:26.270308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:26.269878    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:26.270308 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:26.270004    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:26.270308 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:26.270076    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:26.270308 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:26.269878    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:26.270308 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:26.270159    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:28.270168 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:28.270128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:28.270605 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:28.270128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:28.270605 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:28.270250    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:28.270605 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:28.270128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:28.270605 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:28.270322    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:28.270605 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:28.270438    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:30.270668 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:30.270634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:30.271126 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:30.270729    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:30.271126 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:30.270751    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:30.271126 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:30.270817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:30.271126 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:30.270930    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:30.271126 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:30.271021    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:32.270611 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:32.270571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:32.271191 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:32.270578    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:32.271191 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:32.270706    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:32.271191 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:32.270737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:32.271191 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:32.270793    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:32.271191 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:32.270847    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:34.270304 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:34.270273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:34.270304 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:34.270273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:34.270826 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:34.270330    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:34.270826 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:34.270426    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:34.270826 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:34.270469    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:34.270826 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:34.270539    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:36.270301 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.270267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:36.270301 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.270296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:36.270822 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.270273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:36.270822 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:36.270380    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knv9v" podUID="67d8b168-67d4-4357-a091-52075e672f5f"
Sep 16 18:44:36.270822 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:36.270459    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rr9t2" podUID="29869605-8fde-45a3-a354-424bafd17eb1"
Sep 16 18:44:36.270822 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:36.270543    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4cwkd" podUID="ed7a87c4-b91c-491e-9c5c-deab72585f4d"
Sep 16 18:44:36.469862 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.469841    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-95.ec2.internal" event="NodeReady"
Sep 16 18:44:36.470020 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.470007    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:44:36.556986 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.556882    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m6wrk"]
Sep 16 18:44:36.558997 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.558980    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m7nbk"]
Sep 16 18:44:36.559129 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.559109    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.560935 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.560914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.561742 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.561721    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:44:36.562394 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.562375    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:44:36.562483 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.562452    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6r5qx\""
Sep 16 18:44:36.562531 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.562483    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:44:36.562734 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.562721    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:44:36.563123 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.563109    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:44:36.563207 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.563192    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:44:36.563320 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.563304    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xrh57\""
Sep 16 18:44:36.579459 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.579438    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m6wrk"]
Sep 16 18:44:36.608062 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.608035    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m7nbk"]
Sep 16 18:44:36.664951 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.664919    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2srkz"]
Sep 16 18:44:36.668117 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.668097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2srkz"
Sep 16 18:44:36.670606 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.670586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:44:36.670886 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.670873    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:44:36.671157 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.671137    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ktvzd\""
Sep 16 18:44:36.671242 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.671142    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:44:36.691882 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.691855    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2srkz"]
Sep 16 18:44:36.745911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.745862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/304d64f3-67ea-4996-bd67-1fc8f9c75a14-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.746073 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.745957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/304d64f3-67ea-4996-bd67-1fc8f9c75a14-data-volume\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.746073 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.745983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppsd\" (UniqueName: \"kubernetes.io/projected/304d64f3-67ea-4996-bd67-1fc8f9c75a14-kube-api-access-wppsd\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.746073 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.746005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/197179e5-e12b-4199-afd0-2789401dfc65-config-volume\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.746073 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.746021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jx27\" (UniqueName: \"kubernetes.io/projected/197179e5-e12b-4199-afd0-2789401dfc65-kube-api-access-2jx27\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.746073 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.746036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/304d64f3-67ea-4996-bd67-1fc8f9c75a14-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.746292 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.746113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/197179e5-e12b-4199-afd0-2789401dfc65-metrics-tls\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.746292 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.746154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197179e5-e12b-4199-afd0-2789401dfc65-tmp-dir\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.746292 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.746183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/304d64f3-67ea-4996-bd67-1fc8f9c75a14-crio-socket\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.846992 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.846915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppsd\" (UniqueName: \"kubernetes.io/projected/304d64f3-67ea-4996-bd67-1fc8f9c75a14-kube-api-access-wppsd\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.846992 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.846959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/197179e5-e12b-4199-afd0-2789401dfc65-config-volume\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.846992 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.846978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jx27\" (UniqueName: \"kubernetes.io/projected/197179e5-e12b-4199-afd0-2789401dfc65-kube-api-access-2jx27\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.847205 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/304d64f3-67ea-4996-bd67-1fc8f9c75a14-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.847205 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/197179e5-e12b-4199-afd0-2789401dfc65-metrics-tls\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.847205 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197179e5-e12b-4199-afd0-2789401dfc65-tmp-dir\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.847205 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/304d64f3-67ea-4996-bd67-1fc8f9c75a14-crio-socket\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.847205 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/304d64f3-67ea-4996-bd67-1fc8f9c75a14-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.847205 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1-cert\") pod \"ingress-canary-2srkz\" (UID: \"c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1\") " pod="openshift-ingress-canary/ingress-canary-2srkz"
Sep 16 18:44:36.847205 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6lg\" (UniqueName: \"kubernetes.io/projected/c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1-kube-api-access-qq6lg\") pod \"ingress-canary-2srkz\" (UID: \"c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1\") " pod="openshift-ingress-canary/ingress-canary-2srkz"
Sep 16 18:44:36.847531 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/304d64f3-67ea-4996-bd67-1fc8f9c75a14-crio-socket\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.847531 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/304d64f3-67ea-4996-bd67-1fc8f9c75a14-data-volume\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.847632 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/304d64f3-67ea-4996-bd67-1fc8f9c75a14-data-volume\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.847632 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/197179e5-e12b-4199-afd0-2789401dfc65-config-volume\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.847632 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/304d64f3-67ea-4996-bd67-1fc8f9c75a14-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.847824 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.847802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197179e5-e12b-4199-afd0-2789401dfc65-tmp-dir\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.851132 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.851116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/197179e5-e12b-4199-afd0-2789401dfc65-metrics-tls\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.851132 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.851123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/304d64f3-67ea-4996-bd67-1fc8f9c75a14-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.858033 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.858014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppsd\" (UniqueName: \"kubernetes.io/projected/304d64f3-67ea-4996-bd67-1fc8f9c75a14-kube-api-access-wppsd\") pod \"insights-runtime-extractor-m6wrk\" (UID: \"304d64f3-67ea-4996-bd67-1fc8f9c75a14\") " pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.861906 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.861874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jx27\" (UniqueName: \"kubernetes.io/projected/197179e5-e12b-4199-afd0-2789401dfc65-kube-api-access-2jx27\") pod \"dns-default-m7nbk\" (UID: \"197179e5-e12b-4199-afd0-2789401dfc65\") " pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.869778 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.869762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m6wrk"
Sep 16 18:44:36.873876 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.873860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:36.948687 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.948634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1-cert\") pod \"ingress-canary-2srkz\" (UID: \"c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1\") " pod="openshift-ingress-canary/ingress-canary-2srkz"
Sep 16 18:44:36.948805 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.948693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6lg\" (UniqueName: \"kubernetes.io/projected/c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1-kube-api-access-qq6lg\") pod \"ingress-canary-2srkz\" (UID: \"c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1\") " pod="openshift-ingress-canary/ingress-canary-2srkz"
Sep 16 18:44:36.951074 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.951055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1-cert\") pod \"ingress-canary-2srkz\" (UID: \"c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1\") " pod="openshift-ingress-canary/ingress-canary-2srkz"
Sep 16 18:44:36.975952 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.975496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6lg\" (UniqueName: \"kubernetes.io/projected/c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1-kube-api-access-qq6lg\") pod \"ingress-canary-2srkz\" (UID: \"c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1\") " pod="openshift-ingress-canary/ingress-canary-2srkz"
Sep 16 18:44:36.976366 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:36.976144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2srkz"
Sep 16 18:44:37.198830 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:37.198797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m7nbk"]
Sep 16 18:44:37.203336 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:37.203304    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197179e5_e12b_4199_afd0_2789401dfc65.slice/crio-4ba042858075739c6b8fdccee544c446e1ed2c10ae359b242b2065a96fed308f WatchSource:0}: Error finding container 4ba042858075739c6b8fdccee544c446e1ed2c10ae359b242b2065a96fed308f: Status 404 returned error can't find the container with id 4ba042858075739c6b8fdccee544c446e1ed2c10ae359b242b2065a96fed308f
Sep 16 18:44:37.206141 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:37.206114    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304d64f3_67ea_4996_bd67_1fc8f9c75a14.slice/crio-0715addf443527943dae4fa21e1822f32cac54b9e2474b22159d0e96aa50624a WatchSource:0}: Error finding container 0715addf443527943dae4fa21e1822f32cac54b9e2474b22159d0e96aa50624a: Status 404 returned error can't find the container with id 0715addf443527943dae4fa21e1822f32cac54b9e2474b22159d0e96aa50624a
Sep 16 18:44:37.206234 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:37.206219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m6wrk"]
Sep 16 18:44:37.270743 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:37.270717    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2srkz"]
Sep 16 18:44:37.274654 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:37.274631    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c4e8a1_73bd_4c00_9cd8_0e4905e590c1.slice/crio-dca6be266629adfaea279bf2e12fdc7d732a292c4598c76ad41d800afa8fe4fa WatchSource:0}: Error finding container dca6be266629adfaea279bf2e12fdc7d732a292c4598c76ad41d800afa8fe4fa: Status 404 returned error can't find the container with id dca6be266629adfaea279bf2e12fdc7d732a292c4598c76ad41d800afa8fe4fa
Sep 16 18:44:37.611661 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:37.611576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2srkz" event={"ID":"c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1","Type":"ContainerStarted","Data":"dca6be266629adfaea279bf2e12fdc7d732a292c4598c76ad41d800afa8fe4fa"}
Sep 16 18:44:37.612775 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:37.612747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7nbk" event={"ID":"197179e5-e12b-4199-afd0-2789401dfc65","Type":"ContainerStarted","Data":"4ba042858075739c6b8fdccee544c446e1ed2c10ae359b242b2065a96fed308f"}
Sep 16 18:44:37.614313 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:37.614294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m6wrk" event={"ID":"304d64f3-67ea-4996-bd67-1fc8f9c75a14","Type":"ContainerStarted","Data":"bcae7c0a31d960a492fa33a1d32c78bb87bc90ef2ab2c6e8768f4e7d2daa79ab"}
Sep 16 18:44:37.614401 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:37.614336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m6wrk" event={"ID":"304d64f3-67ea-4996-bd67-1fc8f9c75a14","Type":"ContainerStarted","Data":"0715addf443527943dae4fa21e1822f32cac54b9e2474b22159d0e96aa50624a"}
Sep 16 18:44:38.270274 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:38.270239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:38.270459 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:38.270423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:38.270676 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:38.270239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:38.275950 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:38.275399    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:44:38.276474 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:38.276450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fq6l6\""
Sep 16 18:44:38.276927 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:38.276906    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:38.276991 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:38.276961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mz62r\""
Sep 16 18:44:38.277208 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:38.277192    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:38.277260 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:38.277205    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:44:39.258959 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.258865    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78cfdddcbf-zqrws"]
Sep 16 18:44:39.261030 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.261004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.263635 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.263612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:44:39.263821 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.263798    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:44:39.264167 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.264152    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:44:39.264783 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.264623    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:44:39.264783 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.264645    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:44:39.264783 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.264626    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:44:39.264783 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.264647    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:44:39.264783 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.264626    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tgftw\""
Sep 16 18:44:39.274122 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.274100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78cfdddcbf-zqrws"]
Sep 16 18:44:39.338267 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.338225    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"]
Sep 16 18:44:39.340486 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.340460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.343017 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.342589    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:44:39.343017 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.342617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:44:39.343017 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.342869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-86zln\""
Sep 16 18:44:39.343746 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.343396    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:44:39.343746 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.343476    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:44:39.343746 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.343599    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:44:39.353276 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.353253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"]
Sep 16 18:44:39.367251 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.367222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-serving-cert\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.367414 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.367284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-config\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.367414 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.367307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-service-ca\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.367414 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.367333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7msm5\" (UniqueName: \"kubernetes.io/projected/60d1c005-fca0-472c-b3a6-390f47c33b1a-kube-api-access-7msm5\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.367414 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.367400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-oauth-config\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.367756 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.367446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-oauth-serving-cert\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.468306 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44x6\" (UniqueName: \"kubernetes.io/projected/3b71e4c7-9555-4ca6-bc67-fe363d681acd-kube-api-access-h44x6\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.468306 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-oauth-config\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.468544 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-oauth-serving-cert\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.468544 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-serving-cert\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.468641 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b71e4c7-9555-4ca6-bc67-fe363d681acd-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.468641 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b71e4c7-9555-4ca6-bc67-fe363d681acd-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.468745 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-config\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.468745 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-service-ca\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.468745 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7msm5\" (UniqueName: \"kubernetes.io/projected/60d1c005-fca0-472c-b3a6-390f47c33b1a-kube-api-access-7msm5\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.468745 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.468706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b71e4c7-9555-4ca6-bc67-fe363d681acd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.469128 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.469082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-oauth-serving-cert\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.469596 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.469572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-service-ca\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.469681 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.469572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-config\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.471266 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.471247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-oauth-config\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.471590 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.471572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-serving-cert\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.479039 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.479012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7msm5\" (UniqueName: \"kubernetes.io/projected/60d1c005-fca0-472c-b3a6-390f47c33b1a-kube-api-access-7msm5\") pod \"console-78cfdddcbf-zqrws\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") " pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.569498 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.569395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b71e4c7-9555-4ca6-bc67-fe363d681acd-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.569498 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.569467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b71e4c7-9555-4ca6-bc67-fe363d681acd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.569727 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.569502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44x6\" (UniqueName: \"kubernetes.io/projected/3b71e4c7-9555-4ca6-bc67-fe363d681acd-kube-api-access-h44x6\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.569727 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.569581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b71e4c7-9555-4ca6-bc67-fe363d681acd-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.570354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.570333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b71e4c7-9555-4ca6-bc67-fe363d681acd-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.571611 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.571587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:39.573006 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.572978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b71e4c7-9555-4ca6-bc67-fe363d681acd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.573375 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.573352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b71e4c7-9555-4ca6-bc67-fe363d681acd-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.581965 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.581944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44x6\" (UniqueName: \"kubernetes.io/projected/3b71e4c7-9555-4ca6-bc67-fe363d681acd-kube-api-access-h44x6\") pod \"prometheus-operator-68bcd7f987-lct5x\" (UID: \"3b71e4c7-9555-4ca6-bc67-fe363d681acd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:39.650992 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:39.650950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"
Sep 16 18:44:40.012334 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.012292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78cfdddcbf-zqrws"]
Sep 16 18:44:40.014489 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.014464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-lct5x"]
Sep 16 18:44:40.016105 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:40.016081    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d1c005_fca0_472c_b3a6_390f47c33b1a.slice/crio-ef909b16478287ea83720a8cca101265464492bc1c554c5a54e67d54c48e8627 WatchSource:0}: Error finding container ef909b16478287ea83720a8cca101265464492bc1c554c5a54e67d54c48e8627: Status 404 returned error can't find the container with id ef909b16478287ea83720a8cca101265464492bc1c554c5a54e67d54c48e8627
Sep 16 18:44:40.018679 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:40.018652    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b71e4c7_9555_4ca6_bc67_fe363d681acd.slice/crio-d255702769a87cd961ea22a89a667cb0b4616ca452251aa396344efba0c38995 WatchSource:0}: Error finding container d255702769a87cd961ea22a89a667cb0b4616ca452251aa396344efba0c38995: Status 404 returned error can't find the container with id d255702769a87cd961ea22a89a667cb0b4616ca452251aa396344efba0c38995
Sep 16 18:44:40.555351 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.554062    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dwh7"
Sep 16 18:44:40.626275 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.626238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2srkz" event={"ID":"c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1","Type":"ContainerStarted","Data":"a0b3d4ee580444df774ffda2ef5e3847f37dcf279a569fe81a9c50d8048911ba"}
Sep 16 18:44:40.629491 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.629398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7nbk" event={"ID":"197179e5-e12b-4199-afd0-2789401dfc65","Type":"ContainerStarted","Data":"9822f7c1ee4a36fd3de24a4904ea5849c5d9894688db67cdc4ded22d2ae4c45a"}
Sep 16 18:44:40.629491 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.629454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7nbk" event={"ID":"197179e5-e12b-4199-afd0-2789401dfc65","Type":"ContainerStarted","Data":"7c977a3981f5d0e5bbf798bd9d49fcd61b9e4f8e4dd5ca2ed8a7ca76f2d783e6"}
Sep 16 18:44:40.629714 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.629687    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:40.631204 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.631094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x" event={"ID":"3b71e4c7-9555-4ca6-bc67-fe363d681acd","Type":"ContainerStarted","Data":"d255702769a87cd961ea22a89a667cb0b4616ca452251aa396344efba0c38995"}
Sep 16 18:44:40.635417 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.635341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m6wrk" event={"ID":"304d64f3-67ea-4996-bd67-1fc8f9c75a14","Type":"ContainerStarted","Data":"60d4dcb1052146192f6fc707a7badfce185f6a4414e903b0fb5523912ee4d260"}
Sep 16 18:44:40.637253 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.637227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cfdddcbf-zqrws" event={"ID":"60d1c005-fca0-472c-b3a6-390f47c33b1a","Type":"ContainerStarted","Data":"ef909b16478287ea83720a8cca101265464492bc1c554c5a54e67d54c48e8627"}
Sep 16 18:44:40.643551 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.643503    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2srkz" podStartSLOduration=2.179718524 podStartE2EDuration="4.643485246s" totalImagesPullingTime="2.463766722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:40.642520234 +0000 UTC m=+60.904197242" watchObservedRunningTime="2026-09-16 18:44:40.643485246 +0000 UTC m=+60.905162260"
Sep 16 18:44:40.668698 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:40.668639    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m7nbk" podStartSLOduration=2.143863707 podStartE2EDuration="4.668613476s" totalImagesPullingTime="2.524749769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:40.66700324 +0000 UTC m=+60.928680253" watchObservedRunningTime="2026-09-16 18:44:40.668613476 +0000 UTC m=+60.930290490"
Sep 16 18:44:42.649325 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:42.649280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x" event={"ID":"3b71e4c7-9555-4ca6-bc67-fe363d681acd","Type":"ContainerStarted","Data":"589b741df5e099f9143dfbbe265ae819d56fae62d1976c6da9c1594a8026e526"}
Sep 16 18:44:42.649786 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:42.649357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x" event={"ID":"3b71e4c7-9555-4ca6-bc67-fe363d681acd","Type":"ContainerStarted","Data":"3ad8465c5bba9f1b487fb9f1dd9abbb6d90ee568fde4cf550cb25d91c76657b7"}
Sep 16 18:44:42.652047 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:42.651965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m6wrk" event={"ID":"304d64f3-67ea-4996-bd67-1fc8f9c75a14","Type":"ContainerStarted","Data":"85507ae64bcd27603a68ae2212bbe2814ed3d2d259527e62d2a5038900fffffa"}
Sep 16 18:44:42.669833 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:42.669780    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-lct5x" podStartSLOduration=1.9906766089999999 podStartE2EDuration="3.669766991s" totalImagesPullingTime="1.679090382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:42.669070625 +0000 UTC m=+62.930747636" watchObservedRunningTime="2026-09-16 18:44:42.669766991 +0000 UTC m=+62.931443997"
Sep 16 18:44:42.700824 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:42.700774    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m6wrk" podStartSLOduration=2.437269739 podStartE2EDuration="6.700754778s" totalImagesPullingTime="4.263485039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:42.699914587 +0000 UTC m=+62.961591600" watchObservedRunningTime="2026-09-16 18:44:42.700754778 +0000 UTC m=+62.962431794"
Sep 16 18:44:43.657032 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:43.656989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cfdddcbf-zqrws" event={"ID":"60d1c005-fca0-472c-b3a6-390f47c33b1a","Type":"ContainerStarted","Data":"5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737"}
Sep 16 18:44:43.682972 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:43.682885    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78cfdddcbf-zqrws" podStartSLOduration=1.779149994 podStartE2EDuration="4.682868636s" totalImagesPullingTime="2.903718642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:43.682536933 +0000 UTC m=+63.944213957" watchObservedRunningTime="2026-09-16 18:44:43.682868636 +0000 UTC m=+63.944545647"
Sep 16 18:44:44.738451 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.738417    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"]
Sep 16 18:44:44.764575 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.764540    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"]
Sep 16 18:44:44.764751 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.764698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:44.768409 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.768383    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:44:44.768530 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.768504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-b5zf6\""
Sep 16 18:44:44.771752 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.771732    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:44:44.780876 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.780859    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t54ww"]
Sep 16 18:44:44.806145 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.806120    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"]
Sep 16 18:44:44.806268 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.806242    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:44.809301 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.809269    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:44:44.809410 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.809333    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rh45j\""
Sep 16 18:44:44.809479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.809437    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:44:44.817307 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.817290    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:44:44.820262 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.820247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:44.822718 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.822701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:44:44.822811 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.822733    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:44:44.823088 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.823071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:44:44.823245 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.823233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qzqzq\""
Sep 16 18:44:44.837158 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.837136    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"]
Sep 16 18:44:44.912828 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.912801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-426cv\" (UniqueName: \"kubernetes.io/projected/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-kube-api-access-426cv\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:44.912828 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.912841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:44.913060 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.912871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-root\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:44.913060 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.912918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-metrics-client-ca\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:44.913060 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.912970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-accelerators-collector-config\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:44.913060 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.912991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:44.913060 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.913008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-textfile\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:44.913060 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.913034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:44.913060 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.913050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-sys\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:44.913413 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.913101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:44.913413 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.913129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-tls\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:44.913413 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.913165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-wtmp\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:44.913413 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:44.913216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gs8n\" (UniqueName: \"kubernetes.io/projected/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-kube-api-access-2gs8n\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014238 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gs8n\" (UniqueName: \"kubernetes.io/projected/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-kube-api-access-2gs8n\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014238 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-426cv\" (UniqueName: \"kubernetes.io/projected/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-kube-api-access-426cv\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.014238 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.014238 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.014572 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/aaa06985-4632-4f9f-bc05-e02d67eaa58c-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.014572 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-root\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014572 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-metrics-client-ca\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014572 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-accelerators-collector-config\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014572 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014572 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-root\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014572 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2k4\" (UniqueName: \"kubernetes.io/projected/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-api-access-dm2k4\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.014572 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-textfile\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014982 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.014982 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-sys\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014982 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.014982 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-tls\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014982 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-sys\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014982 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:45.014907    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 16 18:44:45.014982 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-wtmp\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.014982 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.015387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-metrics-client-ca\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.015387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.015014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa06985-4632-4f9f-bc05-e02d67eaa58c-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.015387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.014916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-textfile\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.015387 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:45.015048    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-openshift-state-metrics-tls podName:c21038a3-a68a-4ab9-b004-ba1aa8c61b1f nodeName:}" failed. No retries permitted until 2026-09-16 18:44:45.515026453 +0000 UTC m=+65.776703443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-c74hj" (UID: "c21038a3-a68a-4ab9-b004-ba1aa8c61b1f") : secret "openshift-state-metrics-tls" not found
Sep 16 18:44:45.015387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.015101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.015387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.015115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.015387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.015271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-wtmp\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.015947 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.015450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-accelerators-collector-config\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.017154 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.017131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.017293 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.017273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-node-exporter-tls\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.017337 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.017321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.023379 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.023354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-426cv\" (UniqueName: \"kubernetes.io/projected/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-kube-api-access-426cv\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.024009 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.023990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gs8n\" (UniqueName: \"kubernetes.io/projected/660f2c28-9fa2-4a58-8b34-b7eb6de0ab31-kube-api-access-2gs8n\") pod \"node-exporter-t54ww\" (UID: \"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31\") " pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.115297 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.115263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t54ww"
Sep 16 18:44:45.115495 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.115471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.115538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.115510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa06985-4632-4f9f-bc05-e02d67eaa58c-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.115640 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:45.115618    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 18:44:45.115685 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.115652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.115728 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:44:45.115694    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-tls podName:aaa06985-4632-4f9f-bc05-e02d67eaa58c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:45.615670466 +0000 UTC m=+65.877347478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-cj7nd" (UID: "aaa06985-4632-4f9f-bc05-e02d67eaa58c") : secret "kube-state-metrics-tls" not found
Sep 16 18:44:45.115787 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.115760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.115837 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.115798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/aaa06985-4632-4f9f-bc05-e02d67eaa58c-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.116000 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.115970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2k4\" (UniqueName: \"kubernetes.io/projected/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-api-access-dm2k4\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.116176 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.116160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/aaa06985-4632-4f9f-bc05-e02d67eaa58c-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.116321 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.116292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa06985-4632-4f9f-bc05-e02d67eaa58c-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.116457 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.116441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.118294 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.118274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.127124 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.127100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2k4\" (UniqueName: \"kubernetes.io/projected/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-api-access-dm2k4\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.143626 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:45.143602    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660f2c28_9fa2_4a58_8b34_b7eb6de0ab31.slice/crio-a0be4446a3484484b695c7573e070da08a3d5ae6f18559030794a8427686f573 WatchSource:0}: Error finding container a0be4446a3484484b695c7573e070da08a3d5ae6f18559030794a8427686f573: Status 404 returned error can't find the container with id a0be4446a3484484b695c7573e070da08a3d5ae6f18559030794a8427686f573
Sep 16 18:44:45.520119 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.520084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.522585 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.522565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21038a3-a68a-4ab9-b004-ba1aa8c61b1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-c74hj\" (UID: \"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.621165 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.621125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.623570 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.623552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa06985-4632-4f9f-bc05-e02d67eaa58c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cj7nd\" (UID: \"aaa06985-4632-4f9f-bc05-e02d67eaa58c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.665524 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.665483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t54ww" event={"ID":"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31","Type":"ContainerStarted","Data":"a0be4446a3484484b695c7573e070da08a3d5ae6f18559030794a8427686f573"}
Sep 16 18:44:45.673668 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.673646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"
Sep 16 18:44:45.729174 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.729076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"
Sep 16 18:44:45.925399 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.925360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:45.927601 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.927576    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:45.938856 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.938828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d8b168-67d4-4357-a091-52075e672f5f-metrics-certs\") pod \"network-metrics-daemon-knv9v\" (UID: \"67d8b168-67d4-4357-a091-52075e672f5f\") " pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:45.974724 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:45.974679    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj"]
Sep 16 18:44:46.018533 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.018497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd"]
Sep 16 18:44:46.026373 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.026344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:46.026508 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.026384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:46.028825 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.028799    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:46.029137 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.029093    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:44:46.039747 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.039683    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:44:46.040292 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.040268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7a87c4-b91c-491e-9c5c-deab72585f4d-original-pull-secret\") pod \"global-pull-secret-syncer-4cwkd\" (UID: \"ed7a87c4-b91c-491e-9c5c-deab72585f4d\") " pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:46.050384 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.050359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9dd\" (UniqueName: \"kubernetes.io/projected/29869605-8fde-45a3-a354-424bafd17eb1-kube-api-access-hh9dd\") pod \"network-check-target-rr9t2\" (UID: \"29869605-8fde-45a3-a354-424bafd17eb1\") " pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:46.089618 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.089579    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mz62r\""
Sep 16 18:44:46.096792 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.096754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fq6l6\""
Sep 16 18:44:46.097736 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.097712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knv9v"
Sep 16 18:44:46.101540 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.101514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4cwkd"
Sep 16 18:44:46.104238 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.104218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:46.141729 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:46.141591    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21038a3_a68a_4ab9_b004_ba1aa8c61b1f.slice/crio-6b63324a963cf8f746ada13624fdeeebc3ddc5d68c878c4dc8fdb4381ac52396 WatchSource:0}: Error finding container 6b63324a963cf8f746ada13624fdeeebc3ddc5d68c878c4dc8fdb4381ac52396: Status 404 returned error can't find the container with id 6b63324a963cf8f746ada13624fdeeebc3ddc5d68c878c4dc8fdb4381ac52396
Sep 16 18:44:46.427148 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.427118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4cwkd"]
Sep 16 18:44:46.430084 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:46.430052    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7a87c4_b91c_491e_9c5c_deab72585f4d.slice/crio-1d6e82750b8c78d34c2a4fb9b4ca2762f1ecbabe4d099c26ab2ba198d0eb74fe WatchSource:0}: Error finding container 1d6e82750b8c78d34c2a4fb9b4ca2762f1ecbabe4d099c26ab2ba198d0eb74fe: Status 404 returned error can't find the container with id 1d6e82750b8c78d34c2a4fb9b4ca2762f1ecbabe4d099c26ab2ba198d0eb74fe
Sep 16 18:44:46.449909 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.449859    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-knv9v"]
Sep 16 18:44:46.452683 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.452662    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rr9t2"]
Sep 16 18:44:46.453281 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:46.453259    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d8b168_67d4_4357_a091_52075e672f5f.slice/crio-9ffc68876eba4291b88acaf82fb28df8d1c46799910a85106ba076514d948090 WatchSource:0}: Error finding container 9ffc68876eba4291b88acaf82fb28df8d1c46799910a85106ba076514d948090: Status 404 returned error can't find the container with id 9ffc68876eba4291b88acaf82fb28df8d1c46799910a85106ba076514d948090
Sep 16 18:44:46.455809 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:46.455785    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29869605_8fde_45a3_a354_424bafd17eb1.slice/crio-5cf8251a546d44916a8754e9870f3777770485f7b9f433a0ef1a75493ca42b76 WatchSource:0}: Error finding container 5cf8251a546d44916a8754e9870f3777770485f7b9f433a0ef1a75493ca42b76: Status 404 returned error can't find the container with id 5cf8251a546d44916a8754e9870f3777770485f7b9f433a0ef1a75493ca42b76
Sep 16 18:44:46.671201 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.671164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-knv9v" event={"ID":"67d8b168-67d4-4357-a091-52075e672f5f","Type":"ContainerStarted","Data":"9ffc68876eba4291b88acaf82fb28df8d1c46799910a85106ba076514d948090"}
Sep 16 18:44:46.672515 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.672485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rr9t2" event={"ID":"29869605-8fde-45a3-a354-424bafd17eb1","Type":"ContainerStarted","Data":"5cf8251a546d44916a8754e9870f3777770485f7b9f433a0ef1a75493ca42b76"}
Sep 16 18:44:46.673996 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.673917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4cwkd" event={"ID":"ed7a87c4-b91c-491e-9c5c-deab72585f4d","Type":"ContainerStarted","Data":"1d6e82750b8c78d34c2a4fb9b4ca2762f1ecbabe4d099c26ab2ba198d0eb74fe"}
Sep 16 18:44:46.676047 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.676014    2580 generic.go:356] "Generic (PLEG): container finished" podID="660f2c28-9fa2-4a58-8b34-b7eb6de0ab31" containerID="3d35b0ff9562406049c1bc9218244ad688ab80f203d51059de2247f261ec8d63" exitCode=0
Sep 16 18:44:46.676339 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.676286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t54ww" event={"ID":"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31","Type":"ContainerDied","Data":"3d35b0ff9562406049c1bc9218244ad688ab80f203d51059de2247f261ec8d63"}
Sep 16 18:44:46.677807 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.677783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd" event={"ID":"aaa06985-4632-4f9f-bc05-e02d67eaa58c","Type":"ContainerStarted","Data":"54a2cc4646ba7442fe5a947a5fc5fb02c6c93fe0c82acbfd9080ade7251bedca"}
Sep 16 18:44:46.680187 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.680157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj" event={"ID":"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f","Type":"ContainerStarted","Data":"471ccb32bfcdd9cc9bf53ad556ee2445f8a1bf8cddbf8356090e423245a789af"}
Sep 16 18:44:46.680261 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.680204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj" event={"ID":"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f","Type":"ContainerStarted","Data":"952beb4b313fc5969cc53ea173a89628b9920617dc7d7a020c6989da042c872c"}
Sep 16 18:44:46.680261 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.680222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj" event={"ID":"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f","Type":"ContainerStarted","Data":"6b63324a963cf8f746ada13624fdeeebc3ddc5d68c878c4dc8fdb4381ac52396"}
Sep 16 18:44:46.756002 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.754263    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"]
Sep 16 18:44:46.757225 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.757202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.759628 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.759605    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vb62p\""
Sep 16 18:44:46.759879 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.759849    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:44:46.760038 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.760013    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-agbnju7japh6e\""
Sep 16 18:44:46.760160 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.760143    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:44:46.760257 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.760242    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:44:46.760381 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.760368    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:44:46.760453 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.760438    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:44:46.778591 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.778564    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"]
Sep 16 18:44:46.835066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.834589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.835066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.834639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2lbc\" (UniqueName: \"kubernetes.io/projected/38bc79e7-4ff8-472c-a51a-23d8c76f5281-kube-api-access-z2lbc\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.835066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.834686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-grpc-tls\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.835066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.834712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38bc79e7-4ff8-472c-a51a-23d8c76f5281-metrics-client-ca\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.835066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.834753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-tls\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.835066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.834800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.835066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.834840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.835066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.834874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.936202 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.936106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-tls\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.936707 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.936206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.936707 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.936251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.936707 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.936298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.936707 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.936358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.936707 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.936388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lbc\" (UniqueName: \"kubernetes.io/projected/38bc79e7-4ff8-472c-a51a-23d8c76f5281-kube-api-access-z2lbc\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.936707 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.936451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-grpc-tls\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.936707 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.936476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38bc79e7-4ff8-472c-a51a-23d8c76f5281-metrics-client-ca\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.937258 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.937210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38bc79e7-4ff8-472c-a51a-23d8c76f5281-metrics-client-ca\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.939992 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.939954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-tls\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.942103 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.941948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.942103 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.942048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.942943 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.942478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.944213 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.944176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.945240 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.945203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/38bc79e7-4ff8-472c-a51a-23d8c76f5281-secret-grpc-tls\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:46.946185 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:46.946162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lbc\" (UniqueName: \"kubernetes.io/projected/38bc79e7-4ff8-472c-a51a-23d8c76f5281-kube-api-access-z2lbc\") pod \"thanos-querier-7d77cbb8bc-p92n6\" (UID: \"38bc79e7-4ff8-472c-a51a-23d8c76f5281\") " pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:47.078112 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:47.078076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:47.386829 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:47.386737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"]
Sep 16 18:44:47.624619 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:47.624584    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bc79e7_4ff8_472c_a51a_23d8c76f5281.slice/crio-1dc635e4acbdc72b84b8a357480ad99eb469462c50db785cee480ffda86367e0 WatchSource:0}: Error finding container 1dc635e4acbdc72b84b8a357480ad99eb469462c50db785cee480ffda86367e0: Status 404 returned error can't find the container with id 1dc635e4acbdc72b84b8a357480ad99eb469462c50db785cee480ffda86367e0
Sep 16 18:44:47.688962 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:47.688829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6" event={"ID":"38bc79e7-4ff8-472c-a51a-23d8c76f5281","Type":"ContainerStarted","Data":"1dc635e4acbdc72b84b8a357480ad99eb469462c50db785cee480ffda86367e0"}
Sep 16 18:44:47.692567 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:47.691735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t54ww" event={"ID":"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31","Type":"ContainerStarted","Data":"0cb1d8b2931c91f4dd85f88da7c594ec60be6c68bbc253d88f0fd7b71c0c2436"}
Sep 16 18:44:47.692567 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:47.691797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t54ww" event={"ID":"660f2c28-9fa2-4a58-8b34-b7eb6de0ab31","Type":"ContainerStarted","Data":"133d06ee7e5a0734eb3b9574b0ed5704c564ea8741d7feada2008d4e3998ddf3"}
Sep 16 18:44:47.715238 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:47.715143    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t54ww" podStartSLOduration=2.613801171 podStartE2EDuration="3.715121168s" totalImagesPullingTime="1.02692075s" totalInitContainerRuntime="74.399247ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:47.714256939 +0000 UTC m=+67.975933949" watchObservedRunningTime="2026-09-16 18:44:47.715121168 +0000 UTC m=+67.976798181"
Sep 16 18:44:48.275568 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.275536    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-778ff567cb-7rsfl"]
Sep 16 18:44:48.278499 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.278474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.285880 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.285824    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778ff567cb-7rsfl"]
Sep 16 18:44:48.289421 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.288465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:44:48.350024 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.349663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-service-ca\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.350024 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.349710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-oauth-config\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.350024 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.349741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7qq\" (UniqueName: \"kubernetes.io/projected/b392eb0f-9b92-4643-847e-3b4baf6f592d-kube-api-access-8z7qq\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.350024 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.349791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-trusted-ca-bundle\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.350024 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.349825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-oauth-serving-cert\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.350024 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.349857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-serving-cert\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.350024 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.349876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-config\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.456911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.452460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-oauth-serving-cert\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.456911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.452512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-serving-cert\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.456911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.452539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-config\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.456911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.452594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-service-ca\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.456911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.452620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-oauth-config\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.456911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.452646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7qq\" (UniqueName: \"kubernetes.io/projected/b392eb0f-9b92-4643-847e-3b4baf6f592d-kube-api-access-8z7qq\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.456911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.453418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-oauth-serving-cert\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.456911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.453558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-trusted-ca-bundle\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.456911 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.454459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-trusted-ca-bundle\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.460690 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.459041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-service-ca\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.460690 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.459574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-config\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.462815 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.462789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-oauth-config\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.466040 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.466016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-serving-cert\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.473850 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.473803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7qq\" (UniqueName: \"kubernetes.io/projected/b392eb0f-9b92-4643-847e-3b4baf6f592d-kube-api-access-8z7qq\") pod \"console-778ff567cb-7rsfl\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") " pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.593076 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.592761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:48.710133 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.709197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd" event={"ID":"aaa06985-4632-4f9f-bc05-e02d67eaa58c","Type":"ContainerStarted","Data":"6cd529b004989c05b869c3e5b07594929ac3d89bf307527fc64fd62c218e2cc0"}
Sep 16 18:44:48.710133 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.709270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd" event={"ID":"aaa06985-4632-4f9f-bc05-e02d67eaa58c","Type":"ContainerStarted","Data":"073f945e7ae68e4a31d94da3d4f3da5dd9517a815d14d5f631f2676e5104d78e"}
Sep 16 18:44:48.710133 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.709306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd" event={"ID":"aaa06985-4632-4f9f-bc05-e02d67eaa58c","Type":"ContainerStarted","Data":"1b3559cc827a54b35d1fc7c47d3c491b5f65b26c3d39aa59bb79fc09046d6b1e"}
Sep 16 18:44:48.715516 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.714846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj" event={"ID":"c21038a3-a68a-4ab9-b004-ba1aa8c61b1f","Type":"ContainerStarted","Data":"84fa2c17142f609c73862c499d1d626f8301cd23fb2b671f4f4cc906549615db"}
Sep 16 18:44:48.720921 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.720805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-knv9v" event={"ID":"67d8b168-67d4-4357-a091-52075e672f5f","Type":"ContainerStarted","Data":"88140fd7a97a67c2de2b446884ec9f7f238f4a14b83d7f2a406a00145d1d6d6a"}
Sep 16 18:44:48.720921 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.720855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-knv9v" event={"ID":"67d8b168-67d4-4357-a091-52075e672f5f","Type":"ContainerStarted","Data":"b763817f0e44cf365a9575742452407cfa931335c817ac4d7e3408981ee170c9"}
Sep 16 18:44:48.738806 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.738711    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cj7nd" podStartSLOduration=2.591426751 podStartE2EDuration="4.738685581s" totalImagesPullingTime="2.14725883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:48.73824697 +0000 UTC m=+68.999923993" watchObservedRunningTime="2026-09-16 18:44:48.738685581 +0000 UTC m=+69.000362594"
Sep 16 18:44:48.767824 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.767716    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-knv9v" podStartSLOduration=66.931493578 podStartE2EDuration="1m8.767695802s" totalImagesPullingTime="1.836202224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:48.765602398 +0000 UTC m=+69.027279411" watchObservedRunningTime="2026-09-16 18:44:48.767695802 +0000 UTC m=+69.029372813"
Sep 16 18:44:48.792077 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.791662    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-c74hj" podStartSLOduration=2.814702494 podStartE2EDuration="4.791639049s" totalImagesPullingTime="1.976936555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:48.79076836 +0000 UTC m=+69.052445373" watchObservedRunningTime="2026-09-16 18:44:48.791639049 +0000 UTC m=+69.053316079"
Sep 16 18:44:48.913384 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:48.913292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778ff567cb-7rsfl"]
Sep 16 18:44:48.917133 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:48.917107    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb392eb0f_9b92_4643_847e_3b4baf6f592d.slice/crio-03100360f7b6130a86547874d0150f71b4c22634a56119ed5cff5022f6b23243 WatchSource:0}: Error finding container 03100360f7b6130a86547874d0150f71b4c22634a56119ed5cff5022f6b23243: Status 404 returned error can't find the container with id 03100360f7b6130a86547874d0150f71b4c22634a56119ed5cff5022f6b23243
Sep 16 18:44:49.165977 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.165622    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77f64d7c89-dsgzg"]
Sep 16 18:44:49.169789 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.169761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.178006 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.177970    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:44:49.178772 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.177993    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:44:49.178772 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.178357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nhb8g\""
Sep 16 18:44:49.178772 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.178107    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:44:49.178772 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.178048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:44:49.178772 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.178650    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cvenlnj0k6mld\""
Sep 16 18:44:49.184406 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.182950    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77f64d7c89-dsgzg"]
Sep 16 18:44:49.262399 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.262366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a31cc2ae-3760-4df3-9360-6fc14b749c30-secret-metrics-server-tls\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.262399 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.262405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a31cc2ae-3760-4df3-9360-6fc14b749c30-secret-metrics-server-client-certs\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.262641 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.262429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a31cc2ae-3760-4df3-9360-6fc14b749c30-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.262641 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.262448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a31cc2ae-3760-4df3-9360-6fc14b749c30-client-ca-bundle\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.262641 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.262477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9tx\" (UniqueName: \"kubernetes.io/projected/a31cc2ae-3760-4df3-9360-6fc14b749c30-kube-api-access-mb9tx\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.262641 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.262500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a31cc2ae-3760-4df3-9360-6fc14b749c30-audit-log\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.262641 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.262528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a31cc2ae-3760-4df3-9360-6fc14b749c30-metrics-server-audit-profiles\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.363440 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.363404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a31cc2ae-3760-4df3-9360-6fc14b749c30-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.363935 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.363459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a31cc2ae-3760-4df3-9360-6fc14b749c30-client-ca-bundle\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.363935 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.363512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9tx\" (UniqueName: \"kubernetes.io/projected/a31cc2ae-3760-4df3-9360-6fc14b749c30-kube-api-access-mb9tx\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.363935 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.363549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a31cc2ae-3760-4df3-9360-6fc14b749c30-audit-log\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.363935 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.363598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a31cc2ae-3760-4df3-9360-6fc14b749c30-metrics-server-audit-profiles\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.363935 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.363638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a31cc2ae-3760-4df3-9360-6fc14b749c30-secret-metrics-server-tls\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.363935 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.363665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a31cc2ae-3760-4df3-9360-6fc14b749c30-secret-metrics-server-client-certs\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.365360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.365303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a31cc2ae-3760-4df3-9360-6fc14b749c30-metrics-server-audit-profiles\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.366043 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.365643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a31cc2ae-3760-4df3-9360-6fc14b749c30-audit-log\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.366043 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.365996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a31cc2ae-3760-4df3-9360-6fc14b749c30-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.367278 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.367257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a31cc2ae-3760-4df3-9360-6fc14b749c30-secret-metrics-server-client-certs\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.368136 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.368091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a31cc2ae-3760-4df3-9360-6fc14b749c30-client-ca-bundle\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.369473 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.369442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a31cc2ae-3760-4df3-9360-6fc14b749c30-secret-metrics-server-tls\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.373104 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.373060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9tx\" (UniqueName: \"kubernetes.io/projected/a31cc2ae-3760-4df3-9360-6fc14b749c30-kube-api-access-mb9tx\") pod \"metrics-server-77f64d7c89-dsgzg\" (UID: \"a31cc2ae-3760-4df3-9360-6fc14b749c30\") " pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.488501 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.488464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:44:49.573048 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.573012    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:49.573662 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.573648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:49.579537 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.579512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:49.727089 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.727041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff567cb-7rsfl" event={"ID":"b392eb0f-9b92-4643-847e-3b4baf6f592d","Type":"ContainerStarted","Data":"32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209"}
Sep 16 18:44:49.727260 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.727111    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff567cb-7rsfl" event={"ID":"b392eb0f-9b92-4643-847e-3b4baf6f592d","Type":"ContainerStarted","Data":"03100360f7b6130a86547874d0150f71b4c22634a56119ed5cff5022f6b23243"}
Sep 16 18:44:49.732299 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.732274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:44:49.748920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.748824    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-778ff567cb-7rsfl" podStartSLOduration=1.748807828 podStartE2EDuration="1.748807828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:49.747336364 +0000 UTC m=+70.009013399" watchObservedRunningTime="2026-09-16 18:44:49.748807828 +0000 UTC m=+70.010484840"
Sep 16 18:44:49.982981 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.982950    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"]
Sep 16 18:44:49.986497 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.986478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:49.991141 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.991117    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:44:49.991665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.991275    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:44:49.991665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.991411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:44:49.991665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.991480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8qbzm\""
Sep 16 18:44:49.991665 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.991509    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:44:49.991966 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.991751    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:44:49.997852 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:49.997831    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:44:50.003475 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.003406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"]
Sep 16 18:44:50.071782 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.071746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eccd6d44-5310-4e2b-b843-7fb14835e261-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.071782 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.071782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdb9p\" (UniqueName: \"kubernetes.io/projected/eccd6d44-5310-4e2b-b843-7fb14835e261-kube-api-access-tdb9p\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.072036 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.071835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-federate-client-tls\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.072036 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.071856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-secret-telemeter-client\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.072036 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.071874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eccd6d44-5310-4e2b-b843-7fb14835e261-serving-certs-ca-bundle\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.072036 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.071943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eccd6d44-5310-4e2b-b843-7fb14835e261-metrics-client-ca\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.072036 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.071970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-telemeter-client-tls\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.072036 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.071993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.172884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.172842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eccd6d44-5310-4e2b-b843-7fb14835e261-metrics-client-ca\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.173066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.172919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-telemeter-client-tls\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.173066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.172948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.173066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.172978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eccd6d44-5310-4e2b-b843-7fb14835e261-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.173066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.173003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdb9p\" (UniqueName: \"kubernetes.io/projected/eccd6d44-5310-4e2b-b843-7fb14835e261-kube-api-access-tdb9p\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.173066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.173053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-federate-client-tls\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.173320 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.173079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-secret-telemeter-client\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.173320 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.173135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eccd6d44-5310-4e2b-b843-7fb14835e261-serving-certs-ca-bundle\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.173753 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.173724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eccd6d44-5310-4e2b-b843-7fb14835e261-metrics-client-ca\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.173879 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.173780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eccd6d44-5310-4e2b-b843-7fb14835e261-serving-certs-ca-bundle\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.174472 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.174450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eccd6d44-5310-4e2b-b843-7fb14835e261-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.176389 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.176347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.176651 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.176629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-telemeter-client-tls\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.176871 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.176854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-federate-client-tls\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.176996 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.176868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eccd6d44-5310-4e2b-b843-7fb14835e261-secret-telemeter-client\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.182197 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.182177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdb9p\" (UniqueName: \"kubernetes.io/projected/eccd6d44-5310-4e2b-b843-7fb14835e261-kube-api-access-tdb9p\") pod \"telemeter-client-7f4874c55c-xsf6t\" (UID: \"eccd6d44-5310-4e2b-b843-7fb14835e261\") " pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.301195 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.300930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"
Sep 16 18:44:50.981635 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.981596    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:50.985969 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.985942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:50.989325 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.989126    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:44:50.994014 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.993988    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:44:50.994278 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.994257    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k5jgr\""
Sep 16 18:44:50.994350 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.994304    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:44:50.994542 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.994524    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:44:50.994751 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.994735    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:44:50.994848 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.994763    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:44:50.994939 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.994859    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:44:50.995108 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.995004    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:44:50.995172 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.995125    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:44:50.995226 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.995214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:44:50.995360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.995326    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-87dge4uvao1bg\""
Sep 16 18:44:50.995439 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.995394    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:44:50.995882 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.995866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:44:50.996212 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.996190    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:44:50.997054 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:50.996907    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:44:51.003319 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.003298    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:51.080948 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.080886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081136 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.080973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081136 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btb88\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-kube-api-access-btb88\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081136 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081136 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-web-config\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081136 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-config-out\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-config\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081631 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081631 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081631 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081631 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081631 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081631 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.081631 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.081615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.182824 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.182778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.182824 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.182830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183103 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.182859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183103 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.182914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183103 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183103 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183103 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btb88\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-kube-api-access-btb88\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-web-config\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-config-out\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-config\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183759 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183759 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183759 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183759 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.183759 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.184039 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.183954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.184411 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.184103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.184982 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.184601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.184982 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.184824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.186022 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.185529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.187500 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.187442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.188701 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.188678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.189263 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.189244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.189374 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.189279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.189645 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.189619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.190517 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.190085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-web-config\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.190517 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.190191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.190517 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.190481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.191294 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.191254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.191827 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.191788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-config-out\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.192120 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.192096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.192591 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.192570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.193220 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.193191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btb88\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-kube-api-access-btb88\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.193870 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.193847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-config\") pod \"prometheus-k8s-0\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.299997 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.299886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.654861 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:51.654777    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m7nbk"
Sep 16 18:44:52.746257 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.746205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4cwkd" event={"ID":"ed7a87c4-b91c-491e-9c5c-deab72585f4d","Type":"ContainerStarted","Data":"334b6c89581ab8a032c5df624ec820b605278efce07a6ce61869bc55a9b5469c"}
Sep 16 18:44:52.748734 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.748709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6" event={"ID":"38bc79e7-4ff8-472c-a51a-23d8c76f5281","Type":"ContainerStarted","Data":"3c052a1426e3e667d58c612d696b2f1dd570f5641c2891effa14bc2ecd771712"}
Sep 16 18:44:52.748868 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.748769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6" event={"ID":"38bc79e7-4ff8-472c-a51a-23d8c76f5281","Type":"ContainerStarted","Data":"abacc722db1f0bc1b60125fe101f9da13bc5d63cb0ff178f15c001e5f661dcac"}
Sep 16 18:44:52.748868 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.748813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6" event={"ID":"38bc79e7-4ff8-472c-a51a-23d8c76f5281","Type":"ContainerStarted","Data":"a3a3b04ddf2c305e8a7439b7f83479823322948e867696a6de90fab1af2211e2"}
Sep 16 18:44:52.750260 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.750237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rr9t2" event={"ID":"29869605-8fde-45a3-a354-424bafd17eb1","Type":"ContainerStarted","Data":"ea97c800ccf18eaf8a7f3f228743c077ed369ea66a1c7c62fb1ff9ec446d37b4"}
Sep 16 18:44:52.750384 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.750373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:44:52.755448 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.755424    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f4874c55c-xsf6t"]
Sep 16 18:44:52.758141 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:52.758120    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccd6d44_5310_4e2b_b843_7fb14835e261.slice/crio-d0a9f9b677ea5a317842410b493a0ed6868c93ab98fc788f5099bd44093d4a79 WatchSource:0}: Error finding container d0a9f9b677ea5a317842410b493a0ed6868c93ab98fc788f5099bd44093d4a79: Status 404 returned error can't find the container with id d0a9f9b677ea5a317842410b493a0ed6868c93ab98fc788f5099bd44093d4a79
Sep 16 18:44:52.767962 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.767915    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4cwkd" podStartSLOduration=66.730577135 podStartE2EDuration="1m12.767878538s" totalImagesPullingTime="6.037301403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.767242388 +0000 UTC m=+73.028919401" watchObservedRunningTime="2026-09-16 18:44:52.767878538 +0000 UTC m=+73.029555555"
Sep 16 18:44:52.781612 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.781592    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77f64d7c89-dsgzg"]
Sep 16 18:44:52.783630 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:52.783601    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31cc2ae_3760_4df3_9360_6fc14b749c30.slice/crio-7e699c1234b617c78250e3b886bb924b3b648d55752d0fefe1b2092b8feedb35 WatchSource:0}: Error finding container 7e699c1234b617c78250e3b886bb924b3b648d55752d0fefe1b2092b8feedb35: Status 404 returned error can't find the container with id 7e699c1234b617c78250e3b886bb924b3b648d55752d0fefe1b2092b8feedb35
Sep 16 18:44:52.790925 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.790871    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rr9t2" podStartSLOduration=66.782358913 podStartE2EDuration="1m12.790859165s" totalImagesPullingTime="6.008500252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.790743645 +0000 UTC m=+73.052420656" watchObservedRunningTime="2026-09-16 18:44:52.790859165 +0000 UTC m=+73.052536173"
Sep 16 18:44:52.798556 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:52.798535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:52.803002 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:44:52.802980    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42570b9_bc26_4123_9098_40c3a9883cb4.slice/crio-223eeadf5c01a72825774da97f100e1847417336f814c38d0adb29fa6f28b655 WatchSource:0}: Error finding container 223eeadf5c01a72825774da97f100e1847417336f814c38d0adb29fa6f28b655: Status 404 returned error can't find the container with id 223eeadf5c01a72825774da97f100e1847417336f814c38d0adb29fa6f28b655
Sep 16 18:44:53.757454 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:53.757352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t" event={"ID":"eccd6d44-5310-4e2b-b843-7fb14835e261","Type":"ContainerStarted","Data":"d0a9f9b677ea5a317842410b493a0ed6868c93ab98fc788f5099bd44093d4a79"}
Sep 16 18:44:53.759272 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:53.759220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg" event={"ID":"a31cc2ae-3760-4df3-9360-6fc14b749c30","Type":"ContainerStarted","Data":"7e699c1234b617c78250e3b886bb924b3b648d55752d0fefe1b2092b8feedb35"}
Sep 16 18:44:53.760834 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:53.760642    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerStarted","Data":"223eeadf5c01a72825774da97f100e1847417336f814c38d0adb29fa6f28b655"}
Sep 16 18:44:54.308362 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:54.308320    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778ff567cb-7rsfl"]
Sep 16 18:44:55.773541 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.773506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6" event={"ID":"38bc79e7-4ff8-472c-a51a-23d8c76f5281","Type":"ContainerStarted","Data":"54ffbd4d53ae061fe6f7b61d62bf4de1f4fead2b496892193ed1f42e58266927"}
Sep 16 18:44:55.773958 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.773585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6" event={"ID":"38bc79e7-4ff8-472c-a51a-23d8c76f5281","Type":"ContainerStarted","Data":"6dcbaefb5df74ca8c1feb0860208c19fba5515af1e6117121652310bddab7304"}
Sep 16 18:44:55.773958 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.773629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6" event={"ID":"38bc79e7-4ff8-472c-a51a-23d8c76f5281","Type":"ContainerStarted","Data":"d03e59c138bc2fa1b854bfd1e6ff09db2f680d529376d67a371fbcfe437d5353"}
Sep 16 18:44:55.773958 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.773675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:44:55.775873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.775840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t" event={"ID":"eccd6d44-5310-4e2b-b843-7fb14835e261","Type":"ContainerStarted","Data":"92439d7f273d7cbdbd3f6c9659f9bacbc44414ca03c86518178b95f7b63d903c"}
Sep 16 18:44:55.776016 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.775915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t" event={"ID":"eccd6d44-5310-4e2b-b843-7fb14835e261","Type":"ContainerStarted","Data":"c60c978cb53df4f01cd799b9617bb34aeaea1c184da543329dcd5e4056bcff38"}
Sep 16 18:44:55.776016 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.775945    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t" event={"ID":"eccd6d44-5310-4e2b-b843-7fb14835e261","Type":"ContainerStarted","Data":"edf5db6746c790bc0ce01ab9fc65425a4c21b2a5e5461ff4b17f8b26b1c4b45d"}
Sep 16 18:44:55.777478 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.777452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg" event={"ID":"a31cc2ae-3760-4df3-9360-6fc14b749c30","Type":"ContainerStarted","Data":"92ce4cac42082a32d6bc5c216f367cf9be7ba7b0763de4574596b43e995aac48"}
Sep 16 18:44:55.778951 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.778929    2580 generic.go:356] "Generic (PLEG): container finished" podID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerID="481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf" exitCode=0
Sep 16 18:44:55.779036 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.778974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerDied","Data":"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"}
Sep 16 18:44:55.851542 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.851426    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6" podStartSLOduration=2.831834108 podStartE2EDuration="9.851404854s" totalImagesPullingTime="7.019570746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:55.811683906 +0000 UTC m=+76.073360920" watchObservedRunningTime="2026-09-16 18:44:55.851404854 +0000 UTC m=+76.113081868"
Sep 16 18:44:55.852481 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.852449    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f4874c55c-xsf6t" podStartSLOduration=4.738534184 podStartE2EDuration="6.852439999s" totalImagesPullingTime="2.113905815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:55.850256584 +0000 UTC m=+76.111933596" watchObservedRunningTime="2026-09-16 18:44:55.852439999 +0000 UTC m=+76.114117011"
Sep 16 18:44:55.914714 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:55.914668    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg" podStartSLOduration=4.832688527 podStartE2EDuration="6.914652763s" totalImagesPullingTime="2.081964236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:55.913033226 +0000 UTC m=+76.174710239" watchObservedRunningTime="2026-09-16 18:44:55.914652763 +0000 UTC m=+76.176329776"
Sep 16 18:44:58.593914 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:58.593854    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:44:59.807627 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:59.807595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerStarted","Data":"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"}
Sep 16 18:44:59.808000 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:59.807693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerStarted","Data":"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"}
Sep 16 18:44:59.808000 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:59.807722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerStarted","Data":"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"}
Sep 16 18:44:59.808000 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:59.807756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerStarted","Data":"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"}
Sep 16 18:44:59.808000 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:59.807782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerStarted","Data":"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"}
Sep 16 18:44:59.808000 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:59.807805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerStarted","Data":"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"}
Sep 16 18:44:59.838102 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:44:59.837624    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.749213399 podStartE2EDuration="9.837602315s" totalImagesPullingTime="5.060706888s" totalInitContainerRuntime="27.682028ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:59.835554746 +0000 UTC m=+80.097231770" watchObservedRunningTime="2026-09-16 18:44:59.837602315 +0000 UTC m=+80.099279329"
Sep 16 18:45:01.300193 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:01.300046    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:01.790643 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:01.790615    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d77cbb8bc-p92n6"
Sep 16 18:45:06.940886 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:06.940854    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78cfdddcbf-zqrws"]
Sep 16 18:45:09.488730 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:09.488696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:45:09.489161 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:09.488764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:45:19.330093 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.330043    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-778ff567cb-7rsfl" podUID="b392eb0f-9b92-4643-847e-3b4baf6f592d" containerName="console" containerID="cri-o://32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209" gracePeriod=15
Sep 16 18:45:19.580739 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.580683    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:45:19.751498 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.751453    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-serving-cert\") pod \"b392eb0f-9b92-4643-847e-3b4baf6f592d\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") "
Sep 16 18:45:19.751672 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.751555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-oauth-serving-cert\") pod \"b392eb0f-9b92-4643-847e-3b4baf6f592d\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") "
Sep 16 18:45:19.751672 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.751623    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z7qq\" (UniqueName: \"kubernetes.io/projected/b392eb0f-9b92-4643-847e-3b4baf6f592d-kube-api-access-8z7qq\") pod \"b392eb0f-9b92-4643-847e-3b4baf6f592d\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") "
Sep 16 18:45:19.751672 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.751656    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-config\") pod \"b392eb0f-9b92-4643-847e-3b4baf6f592d\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") "
Sep 16 18:45:19.751830 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.751681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-trusted-ca-bundle\") pod \"b392eb0f-9b92-4643-847e-3b4baf6f592d\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") "
Sep 16 18:45:19.751830 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.751713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-oauth-config\") pod \"b392eb0f-9b92-4643-847e-3b4baf6f592d\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") "
Sep 16 18:45:19.751830 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.751748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-service-ca\") pod \"b392eb0f-9b92-4643-847e-3b4baf6f592d\" (UID: \"b392eb0f-9b92-4643-847e-3b4baf6f592d\") "
Sep 16 18:45:19.752108 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.752056    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b392eb0f-9b92-4643-847e-3b4baf6f592d" (UID: "b392eb0f-9b92-4643-847e-3b4baf6f592d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:19.752258 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.752115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-config" (OuterVolumeSpecName: "console-config") pod "b392eb0f-9b92-4643-847e-3b4baf6f592d" (UID: "b392eb0f-9b92-4643-847e-3b4baf6f592d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:19.752258 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.752136    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b392eb0f-9b92-4643-847e-3b4baf6f592d" (UID: "b392eb0f-9b92-4643-847e-3b4baf6f592d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:19.752515 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.752491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-service-ca" (OuterVolumeSpecName: "service-ca") pod "b392eb0f-9b92-4643-847e-3b4baf6f592d" (UID: "b392eb0f-9b92-4643-847e-3b4baf6f592d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:19.754021 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.753998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b392eb0f-9b92-4643-847e-3b4baf6f592d" (UID: "b392eb0f-9b92-4643-847e-3b4baf6f592d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:19.754422 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.754401    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b392eb0f-9b92-4643-847e-3b4baf6f592d" (UID: "b392eb0f-9b92-4643-847e-3b4baf6f592d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:19.754477 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.754454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b392eb0f-9b92-4643-847e-3b4baf6f592d-kube-api-access-8z7qq" (OuterVolumeSpecName: "kube-api-access-8z7qq") pod "b392eb0f-9b92-4643-847e-3b4baf6f592d" (UID: "b392eb0f-9b92-4643-847e-3b4baf6f592d"). InnerVolumeSpecName "kube-api-access-8z7qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:19.852602 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.852519    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-oauth-serving-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:19.852602 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.852552    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z7qq\" (UniqueName: \"kubernetes.io/projected/b392eb0f-9b92-4643-847e-3b4baf6f592d-kube-api-access-8z7qq\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:19.852602 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.852562    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-config\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:19.852602 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.852571    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-trusted-ca-bundle\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:19.852602 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.852579    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-oauth-config\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:19.852602 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.852587    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b392eb0f-9b92-4643-847e-3b4baf6f592d-service-ca\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:19.852602 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.852594    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b392eb0f-9b92-4643-847e-3b4baf6f592d-console-serving-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:19.890939 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.890885    2580 generic.go:356] "Generic (PLEG): container finished" podID="b392eb0f-9b92-4643-847e-3b4baf6f592d" containerID="32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209" exitCode=0
Sep 16 18:45:19.890939 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.890924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff567cb-7rsfl" event={"ID":"b392eb0f-9b92-4643-847e-3b4baf6f592d","Type":"ContainerDied","Data":"32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209"}
Sep 16 18:45:19.891142 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.890947    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff567cb-7rsfl"
Sep 16 18:45:19.891142 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.890980    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff567cb-7rsfl" event={"ID":"b392eb0f-9b92-4643-847e-3b4baf6f592d","Type":"ContainerDied","Data":"03100360f7b6130a86547874d0150f71b4c22634a56119ed5cff5022f6b23243"}
Sep 16 18:45:19.891142 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.891002    2580 scope.go:117] "RemoveContainer" containerID="32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209"
Sep 16 18:45:19.899953 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.899931    2580 scope.go:117] "RemoveContainer" containerID="32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209"
Sep 16 18:45:19.900262 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:45:19.900242    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209\": container with ID starting with 32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209 not found: ID does not exist" containerID="32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209"
Sep 16 18:45:19.900350 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.900276    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209"} err="failed to get container status \"32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209\": rpc error: code = NotFound desc = could not find container \"32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209\": container with ID starting with 32fdf7f4cbf364a96f018eb54c5f7a5b6957e610c77501f10a18c7253a939209 not found: ID does not exist"
Sep 16 18:45:19.912580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.912557    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778ff567cb-7rsfl"]
Sep 16 18:45:19.919939 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:19.919917    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-778ff567cb-7rsfl"]
Sep 16 18:45:20.278074 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:20.276366    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b392eb0f-9b92-4643-847e-3b4baf6f592d" path="/var/lib/kubelet/pods/b392eb0f-9b92-4643-847e-3b4baf6f592d/volumes"
Sep 16 18:45:23.763847 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:23.763815    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rr9t2"
Sep 16 18:45:29.494608 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:29.494578    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:45:29.498479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:29.498453    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77f64d7c89-dsgzg"
Sep 16 18:45:31.962475 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:31.962298    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78cfdddcbf-zqrws" podUID="60d1c005-fca0-472c-b3a6-390f47c33b1a" containerName="console" containerID="cri-o://5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737" gracePeriod=15
Sep 16 18:45:32.214130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.214073    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:45:32.262421 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.262393    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-oauth-serving-cert\") pod \"60d1c005-fca0-472c-b3a6-390f47c33b1a\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") "
Sep 16 18:45:32.262608 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.262447    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-config\") pod \"60d1c005-fca0-472c-b3a6-390f47c33b1a\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") "
Sep 16 18:45:32.262608 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.262466    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7msm5\" (UniqueName: \"kubernetes.io/projected/60d1c005-fca0-472c-b3a6-390f47c33b1a-kube-api-access-7msm5\") pod \"60d1c005-fca0-472c-b3a6-390f47c33b1a\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") "
Sep 16 18:45:32.262608 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.262513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-serving-cert\") pod \"60d1c005-fca0-472c-b3a6-390f47c33b1a\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") "
Sep 16 18:45:32.262777 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.262646    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-oauth-config\") pod \"60d1c005-fca0-472c-b3a6-390f47c33b1a\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") "
Sep 16 18:45:32.262777 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.262687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-service-ca\") pod \"60d1c005-fca0-472c-b3a6-390f47c33b1a\" (UID: \"60d1c005-fca0-472c-b3a6-390f47c33b1a\") "
Sep 16 18:45:32.262879 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.262789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "60d1c005-fca0-472c-b3a6-390f47c33b1a" (UID: "60d1c005-fca0-472c-b3a6-390f47c33b1a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:32.262879 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.262798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-config" (OuterVolumeSpecName: "console-config") pod "60d1c005-fca0-472c-b3a6-390f47c33b1a" (UID: "60d1c005-fca0-472c-b3a6-390f47c33b1a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:32.263025 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.263001    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-oauth-serving-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:32.263085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.263025    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-config\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:32.263085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.263059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-service-ca" (OuterVolumeSpecName: "service-ca") pod "60d1c005-fca0-472c-b3a6-390f47c33b1a" (UID: "60d1c005-fca0-472c-b3a6-390f47c33b1a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:32.265402 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.265369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d1c005-fca0-472c-b3a6-390f47c33b1a-kube-api-access-7msm5" (OuterVolumeSpecName: "kube-api-access-7msm5") pod "60d1c005-fca0-472c-b3a6-390f47c33b1a" (UID: "60d1c005-fca0-472c-b3a6-390f47c33b1a"). InnerVolumeSpecName "kube-api-access-7msm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:32.265542 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.265521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "60d1c005-fca0-472c-b3a6-390f47c33b1a" (UID: "60d1c005-fca0-472c-b3a6-390f47c33b1a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:32.265692 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.265667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "60d1c005-fca0-472c-b3a6-390f47c33b1a" (UID: "60d1c005-fca0-472c-b3a6-390f47c33b1a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:32.364203 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.364168    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-oauth-config\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:32.364203 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.364204    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60d1c005-fca0-472c-b3a6-390f47c33b1a-service-ca\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:32.364203 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.364216    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7msm5\" (UniqueName: \"kubernetes.io/projected/60d1c005-fca0-472c-b3a6-390f47c33b1a-kube-api-access-7msm5\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:32.364498 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.364225    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60d1c005-fca0-472c-b3a6-390f47c33b1a-console-serving-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:45:32.948214 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.948178    2580 generic.go:356] "Generic (PLEG): container finished" podID="60d1c005-fca0-472c-b3a6-390f47c33b1a" containerID="5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737" exitCode=0
Sep 16 18:45:32.948387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.948228    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cfdddcbf-zqrws"
Sep 16 18:45:32.948387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.948247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cfdddcbf-zqrws" event={"ID":"60d1c005-fca0-472c-b3a6-390f47c33b1a","Type":"ContainerDied","Data":"5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737"}
Sep 16 18:45:32.948387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.948296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cfdddcbf-zqrws" event={"ID":"60d1c005-fca0-472c-b3a6-390f47c33b1a","Type":"ContainerDied","Data":"ef909b16478287ea83720a8cca101265464492bc1c554c5a54e67d54c48e8627"}
Sep 16 18:45:32.948387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.948324    2580 scope.go:117] "RemoveContainer" containerID="5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737"
Sep 16 18:45:32.956906 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.956870    2580 scope.go:117] "RemoveContainer" containerID="5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737"
Sep 16 18:45:32.957170 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:45:32.957152    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737\": container with ID starting with 5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737 not found: ID does not exist" containerID="5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737"
Sep 16 18:45:32.957223 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.957179    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737"} err="failed to get container status \"5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737\": rpc error: code = NotFound desc = could not find container \"5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737\": container with ID starting with 5bc54448c3482b917dac3871d7eeffef53dc33624353f46c6fb41f47638a3737 not found: ID does not exist"
Sep 16 18:45:32.966456 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.966429    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78cfdddcbf-zqrws"]
Sep 16 18:45:32.969449 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:32.969422    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78cfdddcbf-zqrws"]
Sep 16 18:45:34.274795 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:34.274755    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d1c005-fca0-472c-b3a6-390f47c33b1a" path="/var/lib/kubelet/pods/60d1c005-fca0-472c-b3a6-390f47c33b1a/volumes"
Sep 16 18:45:51.300283 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:51.300243    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:51.320531 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:51.320504    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:52.034327 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:45:52.034300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:09.383772 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.383739    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:09.384251 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.384066    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="prometheus" containerID="cri-o://803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af" gracePeriod=600
Sep 16 18:46:09.384251 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.384104    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad" gracePeriod=600
Sep 16 18:46:09.384251 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.384173    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="config-reloader" containerID="cri-o://4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a" gracePeriod=600
Sep 16 18:46:09.384251 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.384090    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy" containerID="cri-o://edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70" gracePeriod=600
Sep 16 18:46:09.384492 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.384188    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="thanos-sidecar" containerID="cri-o://39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d" gracePeriod=600
Sep 16 18:46:09.384492 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.384405    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy-web" containerID="cri-o://13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80" gracePeriod=600
Sep 16 18:46:09.639984 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.638390    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:09.677810 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.677767    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-kube-rbac-proxy\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678011 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.677875    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-0\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678011 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.677951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-1\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678011 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.677983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-kubelet-serving-ca-bundle\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678222 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678190    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-tls\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678236    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678252    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-thanos-prometheus-http-client-file\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btb88\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-kube-api-access-btb88\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-metrics-client-certs\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678319    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-metrics-client-ca\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678389    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-trusted-ca-bundle\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678415    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-config\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-2\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-serving-certs-ca-bundle\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-tls-assets\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-grpc-tls\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.678617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678599    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-db\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.679538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-config-out\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.679538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.679538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678688    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-web-config\") pod \"c42570b9-bc26-4123-9098-40c3a9883cb4\" (UID: \"c42570b9-bc26-4123-9098-40c3a9883cb4\") "
Sep 16 18:46:09.679538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:09.679538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.678994    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.679538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.679016    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.679538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.679032    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.679538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.679070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:09.681696 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.681279    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:09.681696 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.681529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:09.682771 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.682747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:09.683316 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.683289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:09.683435 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.683348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:09.683806 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.683763    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-kube-api-access-btb88" (OuterVolumeSpecName: "kube-api-access-btb88") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "kube-api-access-btb88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:09.684471 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.684354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:09.684958 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.684664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:09.685514 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.685477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:09.685616 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.685582    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-config" (OuterVolumeSpecName: "config") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:09.686036 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.686011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-config-out" (OuterVolumeSpecName: "config-out") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:09.686138 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.686125    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:09.686429 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.686388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:09.688168 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.688135    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:09.688168 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.688503    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:09.699418 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.699382    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-web-config" (OuterVolumeSpecName: "web-config") pod "c42570b9-bc26-4123-9098-40c3a9883cb4" (UID: "c42570b9-bc26-4123-9098-40c3a9883cb4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:09.779871 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779831    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btb88\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-kube-api-access-btb88\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.779871 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779863    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-metrics-client-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.779871 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779873    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-metrics-client-ca\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779884    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779926    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-config\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779938    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779948    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779957    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42570b9-bc26-4123-9098-40c3a9883cb4-tls-assets\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779965    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-grpc-tls\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779972    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-db\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779981    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42570b9-bc26-4123-9098-40c3a9883cb4-config-out\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779989    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.779997    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-web-config\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.780005    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-kube-rbac-proxy\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.780013    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42570b9-bc26-4123-9098-40c3a9883cb4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.780020    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.780130 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:09.780030    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42570b9-bc26-4123-9098-40c3a9883cb4-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:46:10.089287 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089249    2580 generic.go:356] "Generic (PLEG): container finished" podID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerID="b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad" exitCode=0
Sep 16 18:46:10.089287 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089279    2580 generic.go:356] "Generic (PLEG): container finished" podID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerID="edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70" exitCode=0
Sep 16 18:46:10.089287 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089285    2580 generic.go:356] "Generic (PLEG): container finished" podID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerID="13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80" exitCode=0
Sep 16 18:46:10.089287 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089291    2580 generic.go:356] "Generic (PLEG): container finished" podID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerID="39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d" exitCode=0
Sep 16 18:46:10.089287 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089295    2580 generic.go:356] "Generic (PLEG): container finished" podID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerID="4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a" exitCode=0
Sep 16 18:46:10.089580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089302    2580 generic.go:356] "Generic (PLEG): container finished" podID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerID="803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af" exitCode=0
Sep 16 18:46:10.089580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089326    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.089580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerDied","Data":"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"}
Sep 16 18:46:10.089580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerDied","Data":"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"}
Sep 16 18:46:10.089580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerDied","Data":"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"}
Sep 16 18:46:10.089580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerDied","Data":"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"}
Sep 16 18:46:10.089580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089477    2580 scope.go:117] "RemoveContainer" containerID="b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"
Sep 16 18:46:10.089580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerDied","Data":"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"}
Sep 16 18:46:10.089580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089544    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerDied","Data":"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"}
Sep 16 18:46:10.090018 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.089586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42570b9-bc26-4123-9098-40c3a9883cb4","Type":"ContainerDied","Data":"223eeadf5c01a72825774da97f100e1847417336f814c38d0adb29fa6f28b655"}
Sep 16 18:46:10.098509 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.098491    2580 scope.go:117] "RemoveContainer" containerID="edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"
Sep 16 18:46:10.106320 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.106304    2580 scope.go:117] "RemoveContainer" containerID="13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"
Sep 16 18:46:10.113244 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.113225    2580 scope.go:117] "RemoveContainer" containerID="39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"
Sep 16 18:46:10.118943 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.118920    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:10.120705 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.120687    2580 scope.go:117] "RemoveContainer" containerID="4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"
Sep 16 18:46:10.125061 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.125040    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:10.128486 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.128468    2580 scope.go:117] "RemoveContainer" containerID="803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"
Sep 16 18:46:10.135720 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.135702    2580 scope.go:117] "RemoveContainer" containerID="481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"
Sep 16 18:46:10.144336 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.144314    2580 scope.go:117] "RemoveContainer" containerID="b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"
Sep 16 18:46:10.144741 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:46:10.144718    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": container with ID starting with b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad not found: ID does not exist" containerID="b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"
Sep 16 18:46:10.144826 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.144752    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"} err="failed to get container status \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": rpc error: code = NotFound desc = could not find container \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": container with ID starting with b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad not found: ID does not exist"
Sep 16 18:46:10.144826 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.144772    2580 scope.go:117] "RemoveContainer" containerID="edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"
Sep 16 18:46:10.145160 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:46:10.145139    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": container with ID starting with edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70 not found: ID does not exist" containerID="edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"
Sep 16 18:46:10.145232 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.145165    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"} err="failed to get container status \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": rpc error: code = NotFound desc = could not find container \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": container with ID starting with edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70 not found: ID does not exist"
Sep 16 18:46:10.145232 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.145183    2580 scope.go:117] "RemoveContainer" containerID="13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"
Sep 16 18:46:10.145487 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:46:10.145469    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": container with ID starting with 13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80 not found: ID does not exist" containerID="13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"
Sep 16 18:46:10.145525 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.145493    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"} err="failed to get container status \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": rpc error: code = NotFound desc = could not find container \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": container with ID starting with 13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80 not found: ID does not exist"
Sep 16 18:46:10.145525 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.145510    2580 scope.go:117] "RemoveContainer" containerID="39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"
Sep 16 18:46:10.145765 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:46:10.145748    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": container with ID starting with 39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d not found: ID does not exist" containerID="39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"
Sep 16 18:46:10.145811 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.145773    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"} err="failed to get container status \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": rpc error: code = NotFound desc = could not find container \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": container with ID starting with 39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d not found: ID does not exist"
Sep 16 18:46:10.145811 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.145793    2580 scope.go:117] "RemoveContainer" containerID="4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"
Sep 16 18:46:10.146085 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:46:10.146070    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": container with ID starting with 4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a not found: ID does not exist" containerID="4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"
Sep 16 18:46:10.146126 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.146100    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"} err="failed to get container status \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": rpc error: code = NotFound desc = could not find container \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": container with ID starting with 4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a not found: ID does not exist"
Sep 16 18:46:10.146126 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.146115    2580 scope.go:117] "RemoveContainer" containerID="803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"
Sep 16 18:46:10.146374 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:46:10.146359    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": container with ID starting with 803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af not found: ID does not exist" containerID="803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"
Sep 16 18:46:10.146412 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.146380    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"} err="failed to get container status \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": rpc error: code = NotFound desc = could not find container \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": container with ID starting with 803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af not found: ID does not exist"
Sep 16 18:46:10.146412 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.146394    2580 scope.go:117] "RemoveContainer" containerID="481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"
Sep 16 18:46:10.146638 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:46:10.146614    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": container with ID starting with 481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf not found: ID does not exist" containerID="481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"
Sep 16 18:46:10.146681 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.146633    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"} err="failed to get container status \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": rpc error: code = NotFound desc = could not find container \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": container with ID starting with 481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf not found: ID does not exist"
Sep 16 18:46:10.146681 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.146648    2580 scope.go:117] "RemoveContainer" containerID="b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"
Sep 16 18:46:10.146919 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.146883    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"} err="failed to get container status \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": rpc error: code = NotFound desc = could not find container \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": container with ID starting with b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad not found: ID does not exist"
Sep 16 18:46:10.146955 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.146921    2580 scope.go:117] "RemoveContainer" containerID="edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"
Sep 16 18:46:10.147133 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.147112    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"} err="failed to get container status \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": rpc error: code = NotFound desc = could not find container \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": container with ID starting with edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70 not found: ID does not exist"
Sep 16 18:46:10.147133 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.147132    2580 scope.go:117] "RemoveContainer" containerID="13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"
Sep 16 18:46:10.147384 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.147368    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"} err="failed to get container status \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": rpc error: code = NotFound desc = could not find container \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": container with ID starting with 13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80 not found: ID does not exist"
Sep 16 18:46:10.147384 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.147384    2580 scope.go:117] "RemoveContainer" containerID="39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"
Sep 16 18:46:10.147625 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.147600    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"} err="failed to get container status \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": rpc error: code = NotFound desc = could not find container \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": container with ID starting with 39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d not found: ID does not exist"
Sep 16 18:46:10.147667 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.147627    2580 scope.go:117] "RemoveContainer" containerID="4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"
Sep 16 18:46:10.147842 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.147825    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"} err="failed to get container status \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": rpc error: code = NotFound desc = could not find container \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": container with ID starting with 4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a not found: ID does not exist"
Sep 16 18:46:10.147884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.147843    2580 scope.go:117] "RemoveContainer" containerID="803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"
Sep 16 18:46:10.148082 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148067    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"} err="failed to get container status \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": rpc error: code = NotFound desc = could not find container \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": container with ID starting with 803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af not found: ID does not exist"
Sep 16 18:46:10.148121 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148082    2580 scope.go:117] "RemoveContainer" containerID="481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"
Sep 16 18:46:10.148300 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148285    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"} err="failed to get container status \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": rpc error: code = NotFound desc = could not find container \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": container with ID starting with 481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf not found: ID does not exist"
Sep 16 18:46:10.148354 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148302    2580 scope.go:117] "RemoveContainer" containerID="b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"
Sep 16 18:46:10.148556 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148541    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"} err="failed to get container status \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": rpc error: code = NotFound desc = could not find container \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": container with ID starting with b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad not found: ID does not exist"
Sep 16 18:46:10.148599 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148556    2580 scope.go:117] "RemoveContainer" containerID="edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"
Sep 16 18:46:10.148751 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148736    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"} err="failed to get container status \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": rpc error: code = NotFound desc = could not find container \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": container with ID starting with edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70 not found: ID does not exist"
Sep 16 18:46:10.148791 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148752    2580 scope.go:117] "RemoveContainer" containerID="13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"
Sep 16 18:46:10.148985 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148969    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"} err="failed to get container status \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": rpc error: code = NotFound desc = could not find container \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": container with ID starting with 13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80 not found: ID does not exist"
Sep 16 18:46:10.149030 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.148986    2580 scope.go:117] "RemoveContainer" containerID="39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"
Sep 16 18:46:10.149184 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149168    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"} err="failed to get container status \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": rpc error: code = NotFound desc = could not find container \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": container with ID starting with 39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d not found: ID does not exist"
Sep 16 18:46:10.149230 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149184    2580 scope.go:117] "RemoveContainer" containerID="4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"
Sep 16 18:46:10.149399 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149383    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"} err="failed to get container status \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": rpc error: code = NotFound desc = could not find container \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": container with ID starting with 4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a not found: ID does not exist"
Sep 16 18:46:10.149399 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149397    2580 scope.go:117] "RemoveContainer" containerID="803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"
Sep 16 18:46:10.149582 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149567    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"} err="failed to get container status \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": rpc error: code = NotFound desc = could not find container \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": container with ID starting with 803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af not found: ID does not exist"
Sep 16 18:46:10.149623 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149581    2580 scope.go:117] "RemoveContainer" containerID="481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"
Sep 16 18:46:10.149759 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149745    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"} err="failed to get container status \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": rpc error: code = NotFound desc = could not find container \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": container with ID starting with 481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf not found: ID does not exist"
Sep 16 18:46:10.149799 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149759    2580 scope.go:117] "RemoveContainer" containerID="b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"
Sep 16 18:46:10.149985 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149969    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"} err="failed to get container status \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": rpc error: code = NotFound desc = could not find container \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": container with ID starting with b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad not found: ID does not exist"
Sep 16 18:46:10.149985 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.149984    2580 scope.go:117] "RemoveContainer" containerID="edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"
Sep 16 18:46:10.150154 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.150135    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"} err="failed to get container status \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": rpc error: code = NotFound desc = could not find container \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": container with ID starting with edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70 not found: ID does not exist"
Sep 16 18:46:10.150197 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.150156    2580 scope.go:117] "RemoveContainer" containerID="13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"
Sep 16 18:46:10.150369 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.150348    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"} err="failed to get container status \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": rpc error: code = NotFound desc = could not find container \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": container with ID starting with 13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80 not found: ID does not exist"
Sep 16 18:46:10.150369 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.150362    2580 scope.go:117] "RemoveContainer" containerID="39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"
Sep 16 18:46:10.150579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.150555    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"} err="failed to get container status \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": rpc error: code = NotFound desc = could not find container \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": container with ID starting with 39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d not found: ID does not exist"
Sep 16 18:46:10.150623 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.150580    2580 scope.go:117] "RemoveContainer" containerID="4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"
Sep 16 18:46:10.150790 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.150776    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"} err="failed to get container status \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": rpc error: code = NotFound desc = could not find container \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": container with ID starting with 4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a not found: ID does not exist"
Sep 16 18:46:10.150838 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.150789    2580 scope.go:117] "RemoveContainer" containerID="803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"
Sep 16 18:46:10.151054 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151038    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"} err="failed to get container status \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": rpc error: code = NotFound desc = could not find container \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": container with ID starting with 803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af not found: ID does not exist"
Sep 16 18:46:10.151054 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151054    2580 scope.go:117] "RemoveContainer" containerID="481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"
Sep 16 18:46:10.151271 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151255    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"} err="failed to get container status \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": rpc error: code = NotFound desc = could not find container \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": container with ID starting with 481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf not found: ID does not exist"
Sep 16 18:46:10.151313 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151273    2580 scope.go:117] "RemoveContainer" containerID="b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"
Sep 16 18:46:10.151474 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151456    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"} err="failed to get container status \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": rpc error: code = NotFound desc = could not find container \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": container with ID starting with b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad not found: ID does not exist"
Sep 16 18:46:10.151474 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151471    2580 scope.go:117] "RemoveContainer" containerID="edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"
Sep 16 18:46:10.151700 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151679    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"} err="failed to get container status \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": rpc error: code = NotFound desc = could not find container \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": container with ID starting with edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70 not found: ID does not exist"
Sep 16 18:46:10.151745 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151703    2580 scope.go:117] "RemoveContainer" containerID="13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"
Sep 16 18:46:10.151964 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151947    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"} err="failed to get container status \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": rpc error: code = NotFound desc = could not find container \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": container with ID starting with 13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80 not found: ID does not exist"
Sep 16 18:46:10.152015 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.151965    2580 scope.go:117] "RemoveContainer" containerID="39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"
Sep 16 18:46:10.152176 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.152160    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"} err="failed to get container status \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": rpc error: code = NotFound desc = could not find container \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": container with ID starting with 39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d not found: ID does not exist"
Sep 16 18:46:10.152215 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.152176    2580 scope.go:117] "RemoveContainer" containerID="4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"
Sep 16 18:46:10.152357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.152341    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"} err="failed to get container status \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": rpc error: code = NotFound desc = could not find container \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": container with ID starting with 4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a not found: ID does not exist"
Sep 16 18:46:10.152406 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.152357    2580 scope.go:117] "RemoveContainer" containerID="803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"
Sep 16 18:46:10.152582 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.152567    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"} err="failed to get container status \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": rpc error: code = NotFound desc = could not find container \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": container with ID starting with 803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af not found: ID does not exist"
Sep 16 18:46:10.152624 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.152583    2580 scope.go:117] "RemoveContainer" containerID="481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"
Sep 16 18:46:10.152823 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.152803    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"} err="failed to get container status \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": rpc error: code = NotFound desc = could not find container \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": container with ID starting with 481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf not found: ID does not exist"
Sep 16 18:46:10.152823 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.152822    2580 scope.go:117] "RemoveContainer" containerID="b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"
Sep 16 18:46:10.153059 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.153043    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad"} err="failed to get container status \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": rpc error: code = NotFound desc = could not find container \"b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad\": container with ID starting with b26c9805753845a09f5a49d4797bb079da00a6510a3b15ce93ae552186e0c6ad not found: ID does not exist"
Sep 16 18:46:10.153099 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.153060    2580 scope.go:117] "RemoveContainer" containerID="edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"
Sep 16 18:46:10.153280 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.153265    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70"} err="failed to get container status \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": rpc error: code = NotFound desc = could not find container \"edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70\": container with ID starting with edddde2cde0d6f3f5bc82622b6d6c009a6a86a73fd442539223d9f5cd4ff4b70 not found: ID does not exist"
Sep 16 18:46:10.153280 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.153280    2580 scope.go:117] "RemoveContainer" containerID="13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"
Sep 16 18:46:10.153504 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.153478    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80"} err="failed to get container status \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": rpc error: code = NotFound desc = could not find container \"13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80\": container with ID starting with 13ce0ddd1b3d4feebf0088922d80063b520bc6539181d8b44924af3ba9b79a80 not found: ID does not exist"
Sep 16 18:46:10.153617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.153505    2580 scope.go:117] "RemoveContainer" containerID="39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"
Sep 16 18:46:10.153833 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.153752    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d"} err="failed to get container status \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": rpc error: code = NotFound desc = could not find container \"39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d\": container with ID starting with 39f4f46632036e2344dae2cd0dd013f516fcb433e2e7658cf7baf723e60b919d not found: ID does not exist"
Sep 16 18:46:10.153951 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.153809    2580 scope.go:117] "RemoveContainer" containerID="4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"
Sep 16 18:46:10.154294 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.154262    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a"} err="failed to get container status \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": rpc error: code = NotFound desc = could not find container \"4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a\": container with ID starting with 4abaa8d45489f2b5479b80fc9715f68a521d0d6d9b00a67cdfd89a7570bd4e8a not found: ID does not exist"
Sep 16 18:46:10.154294 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.154294    2580 scope.go:117] "RemoveContainer" containerID="803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"
Sep 16 18:46:10.154575 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.154550    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af"} err="failed to get container status \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": rpc error: code = NotFound desc = could not find container \"803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af\": container with ID starting with 803b656ab1c2567003632c30832fc02ea98ed011891c275fad1b840e570aa3af not found: ID does not exist"
Sep 16 18:46:10.154659 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.154576    2580 scope.go:117] "RemoveContainer" containerID="481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"
Sep 16 18:46:10.154879 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.154859    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf"} err="failed to get container status \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": rpc error: code = NotFound desc = could not find container \"481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf\": container with ID starting with 481f85874380a2b680252ec89b5d8a909898d0c6a2f235570cf5a2f13a813ddf not found: ID does not exist"
Sep 16 18:46:10.155613 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.155595    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156407    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b392eb0f-9b92-4643-847e-3b4baf6f592d" containerName="console"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156431    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b392eb0f-9b92-4643-847e-3b4baf6f592d" containerName="console"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156456    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156465    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156488    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="prometheus"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156497    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="prometheus"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156509    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="thanos-sidecar"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156518    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="thanos-sidecar"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156526    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy-thanos"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156534    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy-thanos"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156557    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="init-config-reloader"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156565    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="init-config-reloader"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156579    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d1c005-fca0-472c-b3a6-390f47c33b1a" containerName="console"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156587    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d1c005-fca0-472c-b3a6-390f47c33b1a" containerName="console"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156602    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy-web"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156610    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy-web"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156628    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="config-reloader"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156636    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="config-reloader"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156742    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy-thanos"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156753    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60d1c005-fca0-472c-b3a6-390f47c33b1a" containerName="console"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156770    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="thanos-sidecar"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156780    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy-web"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156794    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="kube-rbac-proxy"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156803    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="config-reloader"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156819    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" containerName="prometheus"
Sep 16 18:46:10.159920 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.156826    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b392eb0f-9b92-4643-847e-3b4baf6f592d" containerName="console"
Sep 16 18:46:10.166810 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.166786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.170467 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.170441    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:46:10.170980 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.170952    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:46:10.171041 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.170997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:46:10.171263 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.171247    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:46:10.171341 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.171331    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:46:10.171611 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.171598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:46:10.171685 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.171643    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:46:10.171931 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.171913    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:46:10.172021 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.171979    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:46:10.172126 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.172100    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k5jgr\""
Sep 16 18:46:10.172211 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.172144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:46:10.172279 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.172227    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-87dge4uvao1bg\""
Sep 16 18:46:10.172593 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.172542    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:46:10.173160 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.173144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:46:10.173670 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.173654    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:46:10.183433 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183515 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183515 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183515 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183609 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc8b8cd7-65e6-4049-830b-6ee8b767c637-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183609 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183609 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183697 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc8b8cd7-65e6-4049-830b-6ee8b767c637-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183697 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183697 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183784 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183784 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183784 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183784 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183918 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgz4\" (UniqueName: \"kubernetes.io/projected/bc8b8cd7-65e6-4049-830b-6ee8b767c637-kube-api-access-drgz4\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183918 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-config\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183918 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.183918 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.184044 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.184044 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.183937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.187739 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.187724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:46:10.194229 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.194211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:10.275596 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.275562    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c42570b9-bc26-4123-9098-40c3a9883cb4" path="/var/lib/kubelet/pods/c42570b9-bc26-4123-9098-40c3a9883cb4/volumes"
Sep 16 18:46:10.284616 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.284595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc8b8cd7-65e6-4049-830b-6ee8b767c637-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.284697 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.284625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.284697 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.284648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.284697 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.284689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc8b8cd7-65e6-4049-830b-6ee8b767c637-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.284849 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.284711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.284849 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.284735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.284849 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.284761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.284912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.284971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgz4\" (UniqueName: \"kubernetes.io/projected/bc8b8cd7-65e6-4049-830b-6ee8b767c637-kube-api-access-drgz4\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-config\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.285353 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.286181 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.286181 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.285732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.287336 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.287305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.287987 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.287961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.288226 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.288204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.288312 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.288293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.288406 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.288376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.288458 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.288408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.288650 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.288511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.288650 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.288571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.288977 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.288759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc8b8cd7-65e6-4049-830b-6ee8b767c637-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.288977 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.288877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.289104 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.289081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.289273 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.289250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.290388 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.290368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.290632 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.290610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc8b8cd7-65e6-4049-830b-6ee8b767c637-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.291057 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.291041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-config\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.291206 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.291190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc8b8cd7-65e6-4049-830b-6ee8b767c637-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.291569 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.291552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc8b8cd7-65e6-4049-830b-6ee8b767c637-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.294223 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.294205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgz4\" (UniqueName: \"kubernetes.io/projected/bc8b8cd7-65e6-4049-830b-6ee8b767c637-kube-api-access-drgz4\") pod \"prometheus-k8s-0\" (UID: \"bc8b8cd7-65e6-4049-830b-6ee8b767c637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.477582 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.477554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:10.778095 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:10.778065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:10.780800 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:46:10.780771    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8b8cd7_65e6_4049_830b_6ee8b767c637.slice/crio-761c9fc0f48c1639e7966067ebeb6a61bf9c8b35e5ed7867f5a8080d7f33d87c WatchSource:0}: Error finding container 761c9fc0f48c1639e7966067ebeb6a61bf9c8b35e5ed7867f5a8080d7f33d87c: Status 404 returned error can't find the container with id 761c9fc0f48c1639e7966067ebeb6a61bf9c8b35e5ed7867f5a8080d7f33d87c
Sep 16 18:46:11.095370 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:11.095278    2580 generic.go:356] "Generic (PLEG): container finished" podID="bc8b8cd7-65e6-4049-830b-6ee8b767c637" containerID="ff7599a27e40326fa5b5e7892aa09cf479039a61f8cc7930d4cc33a1f300af9f" exitCode=0
Sep 16 18:46:11.095370 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:11.095314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8b8cd7-65e6-4049-830b-6ee8b767c637","Type":"ContainerDied","Data":"ff7599a27e40326fa5b5e7892aa09cf479039a61f8cc7930d4cc33a1f300af9f"}
Sep 16 18:46:11.095370 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:11.095364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8b8cd7-65e6-4049-830b-6ee8b767c637","Type":"ContainerStarted","Data":"761c9fc0f48c1639e7966067ebeb6a61bf9c8b35e5ed7867f5a8080d7f33d87c"}
Sep 16 18:46:12.103028 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:12.102996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8b8cd7-65e6-4049-830b-6ee8b767c637","Type":"ContainerStarted","Data":"59c12b69306102516874e3528132629bdf56764199bbc5464abe84f4fb3fd6de"}
Sep 16 18:46:12.103369 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:12.103058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8b8cd7-65e6-4049-830b-6ee8b767c637","Type":"ContainerStarted","Data":"bca27c793f8a6dd82a32ee75c363f8d57b13ac9f0c946e6e3b06e2b1afe02369"}
Sep 16 18:46:12.103369 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:12.103083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8b8cd7-65e6-4049-830b-6ee8b767c637","Type":"ContainerStarted","Data":"aa610928f71595b3c16455f65cd0b7a2da0641b150f9a15f20b998b635ba0eb0"}
Sep 16 18:46:12.103369 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:12.103106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8b8cd7-65e6-4049-830b-6ee8b767c637","Type":"ContainerStarted","Data":"e41173dee295d0a4272822eb4129fad24079ef378f327710b98ee83cfd349069"}
Sep 16 18:46:12.103369 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:12.103130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8b8cd7-65e6-4049-830b-6ee8b767c637","Type":"ContainerStarted","Data":"c280ff29b94a1d64c33292698452595a5198c8960dd8bfb295363d02edf1f70d"}
Sep 16 18:46:12.103369 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:12.103166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc8b8cd7-65e6-4049-830b-6ee8b767c637","Type":"ContainerStarted","Data":"4848677baa50e01f2408671ab3939745d144fcef7f47c5ff29f284b7da0196b6"}
Sep 16 18:46:12.132225 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:12.132179    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.114426044 podStartE2EDuration="2.132164297s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.738253ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:12.131073661 +0000 UTC m=+152.392750684" watchObservedRunningTime="2026-09-16 18:46:12.132164297 +0000 UTC m=+152.393841308"
Sep 16 18:46:15.478216 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:46:15.478179    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:10.478686 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:47:10.478642    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:10.494713 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:47:10.494691    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:11.339647 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:47:11.339618    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:40.145161 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:40.145128    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:48:53.087179 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.087144    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4kxd5"]
Sep 16 18:48:53.089222 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.089207    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:48:53.096929 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.092803    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:48:53.096929 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.093271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-v7gzh\""
Sep 16 18:48:53.096929 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.093590    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:48:53.105395 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.105373    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4kxd5"]
Sep 16 18:48:53.155096 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.155060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cf6dad3-eb94-40d1-b2db-9321df26ab19-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4kxd5\" (UID: \"8cf6dad3-eb94-40d1-b2db-9321df26ab19\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:48:53.155242 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.155101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxcrz\" (UniqueName: \"kubernetes.io/projected/8cf6dad3-eb94-40d1-b2db-9321df26ab19-kube-api-access-vxcrz\") pod \"cert-manager-webhook-6486ffd997-4kxd5\" (UID: \"8cf6dad3-eb94-40d1-b2db-9321df26ab19\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:48:53.256249 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.256208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cf6dad3-eb94-40d1-b2db-9321df26ab19-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4kxd5\" (UID: \"8cf6dad3-eb94-40d1-b2db-9321df26ab19\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:48:53.256249 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.256244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxcrz\" (UniqueName: \"kubernetes.io/projected/8cf6dad3-eb94-40d1-b2db-9321df26ab19-kube-api-access-vxcrz\") pod \"cert-manager-webhook-6486ffd997-4kxd5\" (UID: \"8cf6dad3-eb94-40d1-b2db-9321df26ab19\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:48:53.265319 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.265291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cf6dad3-eb94-40d1-b2db-9321df26ab19-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4kxd5\" (UID: \"8cf6dad3-eb94-40d1-b2db-9321df26ab19\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:48:53.265473 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.265453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxcrz\" (UniqueName: \"kubernetes.io/projected/8cf6dad3-eb94-40d1-b2db-9321df26ab19-kube-api-access-vxcrz\") pod \"cert-manager-webhook-6486ffd997-4kxd5\" (UID: \"8cf6dad3-eb94-40d1-b2db-9321df26ab19\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:48:53.409572 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.409484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:48:53.683664 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.683639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4kxd5"]
Sep 16 18:48:53.686219 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:48:53.686183    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cf6dad3_eb94_40d1_b2db_9321df26ab19.slice/crio-8df502ee77964e38129df61258e150cb09db73b8218d13db6bcdcb773d81e0c3 WatchSource:0}: Error finding container 8df502ee77964e38129df61258e150cb09db73b8218d13db6bcdcb773d81e0c3: Status 404 returned error can't find the container with id 8df502ee77964e38129df61258e150cb09db73b8218d13db6bcdcb773d81e0c3
Sep 16 18:48:53.688230 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.688207    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:48:53.714519 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:53.714494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5" event={"ID":"8cf6dad3-eb94-40d1-b2db-9321df26ab19","Type":"ContainerStarted","Data":"8df502ee77964e38129df61258e150cb09db73b8218d13db6bcdcb773d81e0c3"}
Sep 16 18:48:54.524179 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.524139    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"]
Sep 16 18:48:54.526722 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.526697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"
Sep 16 18:48:54.529203 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.529123    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-q2pfn\""
Sep 16 18:48:54.536470 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.536358    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"]
Sep 16 18:48:54.668753 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.668720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82f2b5ab-fce4-4809-b337-bf23426b6717-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z5g9g\" (UID: \"82f2b5ab-fce4-4809-b337-bf23426b6717\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"
Sep 16 18:48:54.668967 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.668866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpt2\" (UniqueName: \"kubernetes.io/projected/82f2b5ab-fce4-4809-b337-bf23426b6717-kube-api-access-8xpt2\") pod \"cert-manager-cainjector-559cdd8c87-z5g9g\" (UID: \"82f2b5ab-fce4-4809-b337-bf23426b6717\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"
Sep 16 18:48:54.770039 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.770000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82f2b5ab-fce4-4809-b337-bf23426b6717-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z5g9g\" (UID: \"82f2b5ab-fce4-4809-b337-bf23426b6717\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"
Sep 16 18:48:54.770262 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.770084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpt2\" (UniqueName: \"kubernetes.io/projected/82f2b5ab-fce4-4809-b337-bf23426b6717-kube-api-access-8xpt2\") pod \"cert-manager-cainjector-559cdd8c87-z5g9g\" (UID: \"82f2b5ab-fce4-4809-b337-bf23426b6717\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"
Sep 16 18:48:54.783010 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.782801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpt2\" (UniqueName: \"kubernetes.io/projected/82f2b5ab-fce4-4809-b337-bf23426b6717-kube-api-access-8xpt2\") pod \"cert-manager-cainjector-559cdd8c87-z5g9g\" (UID: \"82f2b5ab-fce4-4809-b337-bf23426b6717\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"
Sep 16 18:48:54.783133 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.783088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82f2b5ab-fce4-4809-b337-bf23426b6717-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z5g9g\" (UID: \"82f2b5ab-fce4-4809-b337-bf23426b6717\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"
Sep 16 18:48:54.840312 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:54.840277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"
Sep 16 18:48:56.705934 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:56.705870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g"]
Sep 16 18:48:56.708566 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:48:56.708539    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82f2b5ab_fce4_4809_b337_bf23426b6717.slice/crio-c3bac7e0e4f5db3e8861611c673e42a8c9993a48c2d11323627a9d63f10bcf36 WatchSource:0}: Error finding container c3bac7e0e4f5db3e8861611c673e42a8c9993a48c2d11323627a9d63f10bcf36: Status 404 returned error can't find the container with id c3bac7e0e4f5db3e8861611c673e42a8c9993a48c2d11323627a9d63f10bcf36
Sep 16 18:48:56.729376 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:56.729352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5" event={"ID":"8cf6dad3-eb94-40d1-b2db-9321df26ab19","Type":"ContainerStarted","Data":"56a40bdd9cc1da7d88747af7aaea07fe05c6113a88bbf12672bb47cfb52c9c98"}
Sep 16 18:48:56.729489 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:56.729418    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:48:56.730691 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:56.730662    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g" event={"ID":"82f2b5ab-fce4-4809-b337-bf23426b6717","Type":"ContainerStarted","Data":"c3bac7e0e4f5db3e8861611c673e42a8c9993a48c2d11323627a9d63f10bcf36"}
Sep 16 18:48:56.747191 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:56.747143    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5" podStartSLOduration=0.959714024 podStartE2EDuration="3.74712454s" totalImagesPullingTime="2.787410516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:56.745999956 +0000 UTC m=+317.007676967" watchObservedRunningTime="2026-09-16 18:48:56.74712454 +0000 UTC m=+317.008801553"
Sep 16 18:48:57.735859 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:57.735818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g" event={"ID":"82f2b5ab-fce4-4809-b337-bf23426b6717","Type":"ContainerStarted","Data":"c94a842ad840cef784886e84c4995ac2e7871f00b28c399bd2a896b6a09812e0"}
Sep 16 18:48:57.765150 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:48:57.765104    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z5g9g" podStartSLOduration=3.765089741 podStartE2EDuration="3.765089741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:57.763299562 +0000 UTC m=+318.024976568" watchObservedRunningTime="2026-09-16 18:48:57.765089741 +0000 UTC m=+318.026766754"
Sep 16 18:49:02.738401 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:02.738365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4kxd5"
Sep 16 18:49:10.378165 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.378121    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"]
Sep 16 18:49:10.386129 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.386103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"
Sep 16 18:49:10.389823 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.389800    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:49:10.389927 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.389824    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:49:10.394957 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.394939    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fxd9f\""
Sep 16 18:49:10.395947 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.395921    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"]
Sep 16 18:49:10.502559 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.502534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrhl\" (UniqueName: \"kubernetes.io/projected/e5f79e12-d03f-4f2a-8e20-51124afe58c7-kube-api-access-ltrhl\") pod \"openshift-lws-operator-fd8ccff4c-mps75\" (UID: \"e5f79e12-d03f-4f2a-8e20-51124afe58c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"
Sep 16 18:49:10.502732 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.502594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5f79e12-d03f-4f2a-8e20-51124afe58c7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mps75\" (UID: \"e5f79e12-d03f-4f2a-8e20-51124afe58c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"
Sep 16 18:49:10.603923 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.603873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrhl\" (UniqueName: \"kubernetes.io/projected/e5f79e12-d03f-4f2a-8e20-51124afe58c7-kube-api-access-ltrhl\") pod \"openshift-lws-operator-fd8ccff4c-mps75\" (UID: \"e5f79e12-d03f-4f2a-8e20-51124afe58c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"
Sep 16 18:49:10.604083 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.603935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5f79e12-d03f-4f2a-8e20-51124afe58c7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mps75\" (UID: \"e5f79e12-d03f-4f2a-8e20-51124afe58c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"
Sep 16 18:49:10.604220 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.604202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5f79e12-d03f-4f2a-8e20-51124afe58c7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mps75\" (UID: \"e5f79e12-d03f-4f2a-8e20-51124afe58c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"
Sep 16 18:49:10.612759 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.612732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrhl\" (UniqueName: \"kubernetes.io/projected/e5f79e12-d03f-4f2a-8e20-51124afe58c7-kube-api-access-ltrhl\") pod \"openshift-lws-operator-fd8ccff4c-mps75\" (UID: \"e5f79e12-d03f-4f2a-8e20-51124afe58c7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"
Sep 16 18:49:10.695651 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.695631    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"
Sep 16 18:49:10.965372 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:10.965314    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75"]
Sep 16 18:49:10.967813 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:49:10.967789    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5f79e12_d03f_4f2a_8e20_51124afe58c7.slice/crio-fdc4d3e93a4856fda092c6206b3c88d247d5251d5c85085b9a3b87c32455c075 WatchSource:0}: Error finding container fdc4d3e93a4856fda092c6206b3c88d247d5251d5c85085b9a3b87c32455c075: Status 404 returned error can't find the container with id fdc4d3e93a4856fda092c6206b3c88d247d5251d5c85085b9a3b87c32455c075
Sep 16 18:49:11.791609 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:11.791572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75" event={"ID":"e5f79e12-d03f-4f2a-8e20-51124afe58c7","Type":"ContainerStarted","Data":"fdc4d3e93a4856fda092c6206b3c88d247d5251d5c85085b9a3b87c32455c075"}
Sep 16 18:49:13.801960 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:13.801865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75" event={"ID":"e5f79e12-d03f-4f2a-8e20-51124afe58c7","Type":"ContainerStarted","Data":"f564ce0f27250221683f14211ca95d33690a92318706c65ba095434e71ae0e90"}
Sep 16 18:49:13.825533 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:13.825464    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mps75" podStartSLOduration=1.315044292 podStartE2EDuration="3.825448041s" totalImagesPullingTime="2.510403749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:13.823613977 +0000 UTC m=+334.085290989" watchObservedRunningTime="2026-09-16 18:49:13.825448041 +0000 UTC m=+334.087125053"
Sep 16 18:49:37.108034 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.107996    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"]
Sep 16 18:49:37.110445 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.110423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.113975 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.113951    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hjk7s\""
Sep 16 18:49:37.114082 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.113960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:49:37.116092 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.116076    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:49:37.117110 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.117093    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:49:37.133478 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.133460    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"]
Sep 16 18:49:37.216256 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.216224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvln\" (UniqueName: \"kubernetes.io/projected/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-kube-api-access-6qvln\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.216256 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.216259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-cert\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.216436 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.216289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-metrics-cert\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.216436 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.216345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-manager-config\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.317086 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.317051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-cert\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.317253 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.317108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-metrics-cert\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.317253 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.317150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-manager-config\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.317253 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.317208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvln\" (UniqueName: \"kubernetes.io/projected/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-kube-api-access-6qvln\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.317847 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.317768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-manager-config\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.319692 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.319670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-cert\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.319794 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.319710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-metrics-cert\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.334381 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.334354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvln\" (UniqueName: \"kubernetes.io/projected/65d406dc-ed70-4c70-aa5e-305c5d9d8c16-kube-api-access-6qvln\") pod \"lws-controller-manager-77db69dd7c-f4tg4\" (UID: \"65d406dc-ed70-4c70-aa5e-305c5d9d8c16\") " pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.419123 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.419095    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:37.737008 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.736977    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"]
Sep 16 18:49:37.739490 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:49:37.739463    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d406dc_ed70_4c70_aa5e_305c5d9d8c16.slice/crio-496abb20d0fa650140f9287361543c13f006503cb3f468331a976e84fb9028f7 WatchSource:0}: Error finding container 496abb20d0fa650140f9287361543c13f006503cb3f468331a976e84fb9028f7: Status 404 returned error can't find the container with id 496abb20d0fa650140f9287361543c13f006503cb3f468331a976e84fb9028f7
Sep 16 18:49:37.895661 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:37.895628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4" event={"ID":"65d406dc-ed70-4c70-aa5e-305c5d9d8c16","Type":"ContainerStarted","Data":"496abb20d0fa650140f9287361543c13f006503cb3f468331a976e84fb9028f7"}
Sep 16 18:49:39.907249 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:39.907220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4" event={"ID":"65d406dc-ed70-4c70-aa5e-305c5d9d8c16","Type":"ContainerStarted","Data":"b4fd6d2f842fece9fa7af09ebd6ad7e25fc7579f51c29ffbb686e71924be0f16"}
Sep 16 18:49:39.907629 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:39.907278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:49:39.953100 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:39.953048    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4" podStartSLOduration=1.302846772 podStartE2EDuration="2.953033057s" totalImagesPullingTime="1.650186285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:39.952411278 +0000 UTC m=+360.214088290" watchObservedRunningTime="2026-09-16 18:49:39.953033057 +0000 UTC m=+360.214710069"
Sep 16 18:49:50.914907 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:49:50.914870    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-77db69dd7c-f4tg4"
Sep 16 18:50:13.990252 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:13.990212    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"]
Sep 16 18:50:13.992827 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:13.992805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"
Sep 16 18:50:13.996107 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:13.996086    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-crm2m\""
Sep 16 18:50:13.996225 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:13.996126    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:50:13.996443 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:13.996429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:50:14.014121 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:14.014094    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"]
Sep 16 18:50:14.123303 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:14.123270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvsr\" (UniqueName: \"kubernetes.io/projected/9eea2e73-cb24-4d53-95f0-0b5fdc3b9648-kube-api-access-7nvsr\") pod \"authorino-operator-6fc8f864f4-kqh6t\" (UID: \"9eea2e73-cb24-4d53-95f0-0b5fdc3b9648\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"
Sep 16 18:50:14.224645 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:14.224611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvsr\" (UniqueName: \"kubernetes.io/projected/9eea2e73-cb24-4d53-95f0-0b5fdc3b9648-kube-api-access-7nvsr\") pod \"authorino-operator-6fc8f864f4-kqh6t\" (UID: \"9eea2e73-cb24-4d53-95f0-0b5fdc3b9648\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"
Sep 16 18:50:14.242298 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:14.242246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvsr\" (UniqueName: \"kubernetes.io/projected/9eea2e73-cb24-4d53-95f0-0b5fdc3b9648-kube-api-access-7nvsr\") pod \"authorino-operator-6fc8f864f4-kqh6t\" (UID: \"9eea2e73-cb24-4d53-95f0-0b5fdc3b9648\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"
Sep 16 18:50:14.302265 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:14.302221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"
Sep 16 18:50:14.607630 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:14.607609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"]
Sep 16 18:50:14.609799 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:50:14.609775    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eea2e73_cb24_4d53_95f0_0b5fdc3b9648.slice/crio-cc6b151ff4609cb032033f9e751441897bb7d4ae4e5cba553081a094e743e336 WatchSource:0}: Error finding container cc6b151ff4609cb032033f9e751441897bb7d4ae4e5cba553081a094e743e336: Status 404 returned error can't find the container with id cc6b151ff4609cb032033f9e751441897bb7d4ae4e5cba553081a094e743e336
Sep 16 18:50:15.046101 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:15.046069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t" event={"ID":"9eea2e73-cb24-4d53-95f0-0b5fdc3b9648","Type":"ContainerStarted","Data":"cc6b151ff4609cb032033f9e751441897bb7d4ae4e5cba553081a094e743e336"}
Sep 16 18:50:16.132624 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:16.132583    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"]
Sep 16 18:50:16.136326 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:16.136299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"
Sep 16 18:50:16.139087 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:16.139027    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 18:50:16.139087 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:16.139064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bwph9\""
Sep 16 18:50:16.150739 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:16.150711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"]
Sep 16 18:50:16.242261 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:16.242219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgp66\" (UniqueName: \"kubernetes.io/projected/e64ece76-0bf8-4a9f-95eb-7869c05d70f7-kube-api-access-mgp66\") pod \"dns-operator-controller-manager-65dc8f76f5-svjrf\" (UID: \"e64ece76-0bf8-4a9f-95eb-7869c05d70f7\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"
Sep 16 18:50:16.343060 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:16.343031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgp66\" (UniqueName: \"kubernetes.io/projected/e64ece76-0bf8-4a9f-95eb-7869c05d70f7-kube-api-access-mgp66\") pod \"dns-operator-controller-manager-65dc8f76f5-svjrf\" (UID: \"e64ece76-0bf8-4a9f-95eb-7869c05d70f7\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"
Sep 16 18:50:16.353086 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:16.353062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgp66\" (UniqueName: \"kubernetes.io/projected/e64ece76-0bf8-4a9f-95eb-7869c05d70f7-kube-api-access-mgp66\") pod \"dns-operator-controller-manager-65dc8f76f5-svjrf\" (UID: \"e64ece76-0bf8-4a9f-95eb-7869c05d70f7\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"
Sep 16 18:50:16.450370 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:16.450341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"
Sep 16 18:50:17.297970 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:17.297937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"]
Sep 16 18:50:17.300269 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:50:17.300243    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64ece76_0bf8_4a9f_95eb_7869c05d70f7.slice/crio-c967e6268b528f03b51ee442b02490be873cb8efe342c83538bd55fab5146e52 WatchSource:0}: Error finding container c967e6268b528f03b51ee442b02490be873cb8efe342c83538bd55fab5146e52: Status 404 returned error can't find the container with id c967e6268b528f03b51ee442b02490be873cb8efe342c83538bd55fab5146e52
Sep 16 18:50:17.987009 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:17.986980    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"]
Sep 16 18:50:17.989168 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:17.989154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:17.991656 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:17.991635    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 18:50:17.991795 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:17.991690    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 18:50:17.991795 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:17.991704    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zj67w\""
Sep 16 18:50:18.004316 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:17.999943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"]
Sep 16 18:50:18.060980 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.060947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf" event={"ID":"e64ece76-0bf8-4a9f-95eb-7869c05d70f7","Type":"ContainerStarted","Data":"c967e6268b528f03b51ee442b02490be873cb8efe342c83538bd55fab5146e52"}
Sep 16 18:50:18.062579 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.062553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t" event={"ID":"9eea2e73-cb24-4d53-95f0-0b5fdc3b9648","Type":"ContainerStarted","Data":"ea6769b8cc0e796bfe5b4bdbfc56db201db2ee42b015609701641fdec5ff269a"}
Sep 16 18:50:18.062720 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.062701    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"
Sep 16 18:50:18.082538 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.082495    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t" podStartSLOduration=2.636000343 podStartE2EDuration="5.082483002s" totalImagesPullingTime="2.446482659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:18.080718061 +0000 UTC m=+398.342395073" watchObservedRunningTime="2026-09-16 18:50:18.082483002 +0000 UTC m=+398.344160014"
Sep 16 18:50:18.160831 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.160803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e3165ef0-0332-4646-acc9-05aa0b398780-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.160976 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.160849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nhx\" (UniqueName: \"kubernetes.io/projected/e3165ef0-0332-4646-acc9-05aa0b398780-kube-api-access-h4nhx\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.160976 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.160873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3165ef0-0332-4646-acc9-05aa0b398780-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.262003 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.261927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e3165ef0-0332-4646-acc9-05aa0b398780-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.262003 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.261977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nhx\" (UniqueName: \"kubernetes.io/projected/e3165ef0-0332-4646-acc9-05aa0b398780-kube-api-access-h4nhx\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.262003 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.262002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3165ef0-0332-4646-acc9-05aa0b398780-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.262239 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:50:18.262104    2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 18:50:18.262239 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:50:18.262165    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3165ef0-0332-4646-acc9-05aa0b398780-plugin-serving-cert podName:e3165ef0-0332-4646-acc9-05aa0b398780 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:18.76214611 +0000 UTC m=+399.023823101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/e3165ef0-0332-4646-acc9-05aa0b398780-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-92gtp" (UID: "e3165ef0-0332-4646-acc9-05aa0b398780") : secret "plugin-serving-cert" not found
Sep 16 18:50:18.262640 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.262611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e3165ef0-0332-4646-acc9-05aa0b398780-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.271303 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.271278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nhx\" (UniqueName: \"kubernetes.io/projected/e3165ef0-0332-4646-acc9-05aa0b398780-kube-api-access-h4nhx\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.767594 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.767558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3165ef0-0332-4646-acc9-05aa0b398780-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.770328 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.770304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3165ef0-0332-4646-acc9-05aa0b398780-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-92gtp\" (UID: \"e3165ef0-0332-4646-acc9-05aa0b398780\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:18.898988 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:18.898956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"
Sep 16 18:50:19.280488 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:19.280458    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp"]
Sep 16 18:50:19.283213 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:50:19.283190    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3165ef0_0332_4646_acc9_05aa0b398780.slice/crio-fd14d09d12e090cd4e7726ee89a56440a734459247541f20b6897abf82d7cc35 WatchSource:0}: Error finding container fd14d09d12e090cd4e7726ee89a56440a734459247541f20b6897abf82d7cc35: Status 404 returned error can't find the container with id fd14d09d12e090cd4e7726ee89a56440a734459247541f20b6897abf82d7cc35
Sep 16 18:50:20.075192 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:20.075152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf" event={"ID":"e64ece76-0bf8-4a9f-95eb-7869c05d70f7","Type":"ContainerStarted","Data":"3e353d9bb52104f696de7c79d160b07376a0e5a1ce8509bf193ed12e85512794"}
Sep 16 18:50:20.075636 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:20.075305    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"
Sep 16 18:50:20.076459 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:20.076438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp" event={"ID":"e3165ef0-0332-4646-acc9-05aa0b398780","Type":"ContainerStarted","Data":"fd14d09d12e090cd4e7726ee89a56440a734459247541f20b6897abf82d7cc35"}
Sep 16 18:50:24.098107 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:24.098075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp" event={"ID":"e3165ef0-0332-4646-acc9-05aa0b398780","Type":"ContainerStarted","Data":"e192a6aaac62d16fb135b17b4019cf443c22a6fd5a4e7914098db5e5e132b0ee"}
Sep 16 18:50:24.114516 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:24.114473    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf" podStartSLOduration=6.367662159 podStartE2EDuration="8.114455554s" totalImagesPullingTime="1.746793395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:20.11180932 +0000 UTC m=+400.373486334" watchObservedRunningTime="2026-09-16 18:50:24.114455554 +0000 UTC m=+404.376132558"
Sep 16 18:50:24.114843 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:24.114823    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-92gtp" podStartSLOduration=2.412747053 podStartE2EDuration="7.1148181s" totalImagesPullingTime="4.702071047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:24.113052665 +0000 UTC m=+404.374729677" watchObservedRunningTime="2026-09-16 18:50:24.1148181 +0000 UTC m=+404.376495111"
Sep 16 18:50:29.071634 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:29.071603    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-kqh6t"
Sep 16 18:50:31.084666 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:50:31.084635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-svjrf"
Sep 16 18:51:00.324217 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:00.324187    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5gmts"]
Sep 16 18:51:00.347737 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:00.347706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts"
Sep 16 18:51:00.348010 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:00.347986    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5gmts"]
Sep 16 18:51:00.350999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:00.350974    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l4psz\""
Sep 16 18:51:00.515263 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:00.515233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkb4\" (UniqueName: \"kubernetes.io/projected/e0e25997-0303-4363-82f7-04d8ccf9e9db-kube-api-access-7nkb4\") pod \"authorino-7bcfd8f9cd-5gmts\" (UID: \"e0e25997-0303-4363-82f7-04d8ccf9e9db\") " pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts"
Sep 16 18:51:00.615742 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:00.615662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkb4\" (UniqueName: \"kubernetes.io/projected/e0e25997-0303-4363-82f7-04d8ccf9e9db-kube-api-access-7nkb4\") pod \"authorino-7bcfd8f9cd-5gmts\" (UID: \"e0e25997-0303-4363-82f7-04d8ccf9e9db\") " pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts"
Sep 16 18:51:00.636127 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:00.636097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkb4\" (UniqueName: \"kubernetes.io/projected/e0e25997-0303-4363-82f7-04d8ccf9e9db-kube-api-access-7nkb4\") pod \"authorino-7bcfd8f9cd-5gmts\" (UID: \"e0e25997-0303-4363-82f7-04d8ccf9e9db\") " pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts"
Sep 16 18:51:00.657436 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:00.657411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts"
Sep 16 18:51:00.930057 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:00.930033    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5gmts"]
Sep 16 18:51:00.933272 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:51:00.933242    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0e25997_0303_4363_82f7_04d8ccf9e9db.slice/crio-df88f8de2d770efdca5f873b0573da4a8999e321a2cd5a94ed8c388fa1bc8fa9 WatchSource:0}: Error finding container df88f8de2d770efdca5f873b0573da4a8999e321a2cd5a94ed8c388fa1bc8fa9: Status 404 returned error can't find the container with id df88f8de2d770efdca5f873b0573da4a8999e321a2cd5a94ed8c388fa1bc8fa9
Sep 16 18:51:01.263326 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:01.263237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts" event={"ID":"e0e25997-0303-4363-82f7-04d8ccf9e9db","Type":"ContainerStarted","Data":"df88f8de2d770efdca5f873b0573da4a8999e321a2cd5a94ed8c388fa1bc8fa9"}
Sep 16 18:51:03.274387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:03.274353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts" event={"ID":"e0e25997-0303-4363-82f7-04d8ccf9e9db","Type":"ContainerStarted","Data":"245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759"}
Sep 16 18:51:03.289484 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:03.289441    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts" podStartSLOduration=1.701787556 podStartE2EDuration="3.289425566s" totalImagesPullingTime="1.58763801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:03.288319684 +0000 UTC m=+443.549996699" watchObservedRunningTime="2026-09-16 18:51:03.289425566 +0000 UTC m=+443.551102578"
Sep 16 18:51:04.562782 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:04.562753    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5gmts"]
Sep 16 18:51:05.282495 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:05.282435    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts" podUID="e0e25997-0303-4363-82f7-04d8ccf9e9db" containerName="authorino" containerID="cri-o://245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759" gracePeriod=30
Sep 16 18:51:05.520047 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:05.520025    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts"
Sep 16 18:51:05.661215 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:05.661136    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nkb4\" (UniqueName: \"kubernetes.io/projected/e0e25997-0303-4363-82f7-04d8ccf9e9db-kube-api-access-7nkb4\") pod \"e0e25997-0303-4363-82f7-04d8ccf9e9db\" (UID: \"e0e25997-0303-4363-82f7-04d8ccf9e9db\") "
Sep 16 18:51:05.663437 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:05.663417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e25997-0303-4363-82f7-04d8ccf9e9db-kube-api-access-7nkb4" (OuterVolumeSpecName: "kube-api-access-7nkb4") pod "e0e25997-0303-4363-82f7-04d8ccf9e9db" (UID: "e0e25997-0303-4363-82f7-04d8ccf9e9db"). InnerVolumeSpecName "kube-api-access-7nkb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:05.762672 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:05.762632    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nkb4\" (UniqueName: \"kubernetes.io/projected/e0e25997-0303-4363-82f7-04d8ccf9e9db-kube-api-access-7nkb4\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:51:06.287107 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:06.287075    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0e25997-0303-4363-82f7-04d8ccf9e9db" containerID="245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759" exitCode=0
Sep 16 18:51:06.287252 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:06.287126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts"
Sep 16 18:51:06.287252 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:06.287126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts" event={"ID":"e0e25997-0303-4363-82f7-04d8ccf9e9db","Type":"ContainerDied","Data":"245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759"}
Sep 16 18:51:06.287252 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:06.287235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5gmts" event={"ID":"e0e25997-0303-4363-82f7-04d8ccf9e9db","Type":"ContainerDied","Data":"df88f8de2d770efdca5f873b0573da4a8999e321a2cd5a94ed8c388fa1bc8fa9"}
Sep 16 18:51:06.287401 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:06.287257    2580 scope.go:117] "RemoveContainer" containerID="245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759"
Sep 16 18:51:06.295254 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:06.295231    2580 scope.go:117] "RemoveContainer" containerID="245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759"
Sep 16 18:51:06.295530 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:51:06.295513    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759\": container with ID starting with 245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759 not found: ID does not exist" containerID="245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759"
Sep 16 18:51:06.295581 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:06.295538    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759"} err="failed to get container status \"245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759\": rpc error: code = NotFound desc = could not find container \"245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759\": container with ID starting with 245c76446dc2bc176c1306ffff8208bc495a65d57fdb4ae11ad4fc828e085759 not found: ID does not exist"
Sep 16 18:51:06.302604 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:06.302579    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5gmts"]
Sep 16 18:51:06.306902 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:06.306877    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5gmts"]
Sep 16 18:51:08.275478 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:08.275443    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0e25997-0303-4363-82f7-04d8ccf9e9db" path="/var/lib/kubelet/pods/e0e25997-0303-4363-82f7-04d8ccf9e9db/volumes"
Sep 16 18:51:23.565980 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.565947    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-rttvb"]
Sep 16 18:51:23.566339 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.566279    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e25997-0303-4363-82f7-04d8ccf9e9db" containerName="authorino"
Sep 16 18:51:23.566339 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.566291    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e25997-0303-4363-82f7-04d8ccf9e9db" containerName="authorino"
Sep 16 18:51:23.566405 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.566358    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e25997-0303-4363-82f7-04d8ccf9e9db" containerName="authorino"
Sep 16 18:51:23.569231 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.569209    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-rttvb"
Sep 16 18:51:23.574289 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.574246    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l4psz\""
Sep 16 18:51:23.575002 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.574983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 18:51:23.584777 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.584752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-rttvb"]
Sep 16 18:51:23.609193 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.609164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zt6\" (UniqueName: \"kubernetes.io/projected/8de2c82b-97b2-4cf7-a5e7-8db239b6bdba-kube-api-access-k4zt6\") pod \"authorino-6d9898bd99-rttvb\" (UID: \"8de2c82b-97b2-4cf7-a5e7-8db239b6bdba\") " pod="kuadrant-system/authorino-6d9898bd99-rttvb"
Sep 16 18:51:23.609297 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.609205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8de2c82b-97b2-4cf7-a5e7-8db239b6bdba-tls-cert\") pod \"authorino-6d9898bd99-rttvb\" (UID: \"8de2c82b-97b2-4cf7-a5e7-8db239b6bdba\") " pod="kuadrant-system/authorino-6d9898bd99-rttvb"
Sep 16 18:51:23.715646 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.715550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zt6\" (UniqueName: \"kubernetes.io/projected/8de2c82b-97b2-4cf7-a5e7-8db239b6bdba-kube-api-access-k4zt6\") pod \"authorino-6d9898bd99-rttvb\" (UID: \"8de2c82b-97b2-4cf7-a5e7-8db239b6bdba\") " pod="kuadrant-system/authorino-6d9898bd99-rttvb"
Sep 16 18:51:23.715805 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.715638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8de2c82b-97b2-4cf7-a5e7-8db239b6bdba-tls-cert\") pod \"authorino-6d9898bd99-rttvb\" (UID: \"8de2c82b-97b2-4cf7-a5e7-8db239b6bdba\") " pod="kuadrant-system/authorino-6d9898bd99-rttvb"
Sep 16 18:51:23.718418 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.718395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8de2c82b-97b2-4cf7-a5e7-8db239b6bdba-tls-cert\") pod \"authorino-6d9898bd99-rttvb\" (UID: \"8de2c82b-97b2-4cf7-a5e7-8db239b6bdba\") " pod="kuadrant-system/authorino-6d9898bd99-rttvb"
Sep 16 18:51:23.726178 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.726155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zt6\" (UniqueName: \"kubernetes.io/projected/8de2c82b-97b2-4cf7-a5e7-8db239b6bdba-kube-api-access-k4zt6\") pod \"authorino-6d9898bd99-rttvb\" (UID: \"8de2c82b-97b2-4cf7-a5e7-8db239b6bdba\") " pod="kuadrant-system/authorino-6d9898bd99-rttvb"
Sep 16 18:51:23.878574 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:23.878493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-rttvb"
Sep 16 18:51:24.153021 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:24.152991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-rttvb"]
Sep 16 18:51:24.154371 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:51:24.154345    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8de2c82b_97b2_4cf7_a5e7_8db239b6bdba.slice/crio-478f1db2e3979fdf78f9410248a58aff52b7ab066f455dc32a2f4c1f311c3771 WatchSource:0}: Error finding container 478f1db2e3979fdf78f9410248a58aff52b7ab066f455dc32a2f4c1f311c3771: Status 404 returned error can't find the container with id 478f1db2e3979fdf78f9410248a58aff52b7ab066f455dc32a2f4c1f311c3771
Sep 16 18:51:24.370372 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:24.370335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-rttvb" event={"ID":"8de2c82b-97b2-4cf7-a5e7-8db239b6bdba","Type":"ContainerStarted","Data":"478f1db2e3979fdf78f9410248a58aff52b7ab066f455dc32a2f4c1f311c3771"}
Sep 16 18:51:25.376071 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:25.376040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-rttvb" event={"ID":"8de2c82b-97b2-4cf7-a5e7-8db239b6bdba","Type":"ContainerStarted","Data":"65b7a9ef0879628a1c01cabc22d515454df51a6847dd11d83ef6ddc79a7a1652"}
Sep 16 18:51:25.391547 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:25.391504    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-rttvb" podStartSLOduration=2.00796983 podStartE2EDuration="2.391487977s" totalImagesPullingTime="383.518147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:25.389700936 +0000 UTC m=+465.651377973" watchObservedRunningTime="2026-09-16 18:51:25.391487977 +0000 UTC m=+465.653164997"
Sep 16 18:51:58.347799 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.347763    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-nm8jm"]
Sep 16 18:51:58.350956 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.350922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:51:58.352656 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.352630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nm8jm"]
Sep 16 18:51:58.353380 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.353357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bkns9\""
Sep 16 18:51:58.353380 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.353376    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:51:58.354018 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.353815    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:51:58.354018 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.353878    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:51:58.417314 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.417287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtljl\" (UniqueName: \"kubernetes.io/projected/74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8-kube-api-access-jtljl\") pod \"seaweedfs-64568bcd49-nm8jm\" (UID: \"74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8\") " pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:51:58.417420 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.417341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8-data\") pod \"seaweedfs-64568bcd49-nm8jm\" (UID: \"74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8\") " pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:51:58.517836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.517810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8-data\") pod \"seaweedfs-64568bcd49-nm8jm\" (UID: \"74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8\") " pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:51:58.517999 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.517866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtljl\" (UniqueName: \"kubernetes.io/projected/74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8-kube-api-access-jtljl\") pod \"seaweedfs-64568bcd49-nm8jm\" (UID: \"74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8\") " pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:51:58.518217 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.518200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8-data\") pod \"seaweedfs-64568bcd49-nm8jm\" (UID: \"74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8\") " pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:51:58.526742 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.526721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtljl\" (UniqueName: \"kubernetes.io/projected/74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8-kube-api-access-jtljl\") pod \"seaweedfs-64568bcd49-nm8jm\" (UID: \"74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8\") " pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:51:58.660083 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.660012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:51:58.961705 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:58.961683    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nm8jm"]
Sep 16 18:51:58.964432 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:51:58.964404    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74733630_8c0d_49f5_9b0f_3eaa9bbcdfa8.slice/crio-0f37f0f156d7bf4d38ce4b19420a34b53ea892282593b56e17a0b45a60c99229 WatchSource:0}: Error finding container 0f37f0f156d7bf4d38ce4b19420a34b53ea892282593b56e17a0b45a60c99229: Status 404 returned error can't find the container with id 0f37f0f156d7bf4d38ce4b19420a34b53ea892282593b56e17a0b45a60c99229
Sep 16 18:51:59.524609 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:51:59.524562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nm8jm" event={"ID":"74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8","Type":"ContainerStarted","Data":"0f37f0f156d7bf4d38ce4b19420a34b53ea892282593b56e17a0b45a60c99229"}
Sep 16 18:52:02.541134 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:52:02.541085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nm8jm" event={"ID":"74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8","Type":"ContainerStarted","Data":"56f6b9bbaf949289082fe52b51126b04d13b78745b643c4e47685d8c620bde3c"}
Sep 16 18:52:02.541566 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:52:02.541146    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:52:08.547916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:52:08.547869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-nm8jm"
Sep 16 18:52:08.569577 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:52:08.569530    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-nm8jm" podStartSLOduration=7.857893309 podStartE2EDuration="10.569514212s" totalImagesPullingTime="2.711620903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:02.572542371 +0000 UTC m=+502.834219382" watchObservedRunningTime="2026-09-16 18:52:08.569514212 +0000 UTC m=+508.831191224"
Sep 16 18:53:17.345443 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.345411    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-vrtsf"]
Sep 16 18:53:17.348847 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.348826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:17.351050 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.351028    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:53:17.351215 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.351196    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-zswvr\""
Sep 16 18:53:17.359409 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.359385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vrtsf"]
Sep 16 18:53:17.481737 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.481699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c3ed55-c98f-45d5-bba6-3da076466364-tls-certs\") pod \"model-serving-api-7b954ffc95-vrtsf\" (UID: \"d3c3ed55-c98f-45d5-bba6-3da076466364\") " pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:17.481737 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.481737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qtn\" (UniqueName: \"kubernetes.io/projected/d3c3ed55-c98f-45d5-bba6-3da076466364-kube-api-access-97qtn\") pod \"model-serving-api-7b954ffc95-vrtsf\" (UID: \"d3c3ed55-c98f-45d5-bba6-3da076466364\") " pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:17.582450 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.582423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c3ed55-c98f-45d5-bba6-3da076466364-tls-certs\") pod \"model-serving-api-7b954ffc95-vrtsf\" (UID: \"d3c3ed55-c98f-45d5-bba6-3da076466364\") " pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:17.582450 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.582456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qtn\" (UniqueName: \"kubernetes.io/projected/d3c3ed55-c98f-45d5-bba6-3da076466364-kube-api-access-97qtn\") pod \"model-serving-api-7b954ffc95-vrtsf\" (UID: \"d3c3ed55-c98f-45d5-bba6-3da076466364\") " pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:17.582658 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:53:17.582571    2580 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 18:53:17.582658 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:53:17.582646    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3c3ed55-c98f-45d5-bba6-3da076466364-tls-certs podName:d3c3ed55-c98f-45d5-bba6-3da076466364 nodeName:}" failed. No retries permitted until 2026-09-16 18:53:18.082628275 +0000 UTC m=+578.344305265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d3c3ed55-c98f-45d5-bba6-3da076466364-tls-certs") pod "model-serving-api-7b954ffc95-vrtsf" (UID: "d3c3ed55-c98f-45d5-bba6-3da076466364") : secret "model-serving-api-tls" not found
Sep 16 18:53:17.591122 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:17.591103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qtn\" (UniqueName: \"kubernetes.io/projected/d3c3ed55-c98f-45d5-bba6-3da076466364-kube-api-access-97qtn\") pod \"model-serving-api-7b954ffc95-vrtsf\" (UID: \"d3c3ed55-c98f-45d5-bba6-3da076466364\") " pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:18.087389 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:18.087356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c3ed55-c98f-45d5-bba6-3da076466364-tls-certs\") pod \"model-serving-api-7b954ffc95-vrtsf\" (UID: \"d3c3ed55-c98f-45d5-bba6-3da076466364\") " pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:18.089994 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:18.089970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c3ed55-c98f-45d5-bba6-3da076466364-tls-certs\") pod \"model-serving-api-7b954ffc95-vrtsf\" (UID: \"d3c3ed55-c98f-45d5-bba6-3da076466364\") " pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:18.259401 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:18.259358    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:18.532329 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:18.532291    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vrtsf"]
Sep 16 18:53:18.545155 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:53:18.545112    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c3ed55_c98f_45d5_bba6_3da076466364.slice/crio-ae9a3c5de52ea49f29c6a3be5d42fe0293b76e89c830bcd69fc92acc564828e9 WatchSource:0}: Error finding container ae9a3c5de52ea49f29c6a3be5d42fe0293b76e89c830bcd69fc92acc564828e9: Status 404 returned error can't find the container with id ae9a3c5de52ea49f29c6a3be5d42fe0293b76e89c830bcd69fc92acc564828e9
Sep 16 18:53:18.878460 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:18.878426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vrtsf" event={"ID":"d3c3ed55-c98f-45d5-bba6-3da076466364","Type":"ContainerStarted","Data":"ae9a3c5de52ea49f29c6a3be5d42fe0293b76e89c830bcd69fc92acc564828e9"}
Sep 16 18:53:20.889717 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:20.889629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vrtsf" event={"ID":"d3c3ed55-c98f-45d5-bba6-3da076466364","Type":"ContainerStarted","Data":"8352c531af556c55a22df86ff91b0823f147bb710d4ebddb869d317c388e4455"}
Sep 16 18:53:20.890104 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:20.889757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:53:20.906632 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:20.906588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-vrtsf" podStartSLOduration=1.823466405 podStartE2EDuration="3.906573486s" totalImagesPullingTime="2.083107081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:20.905325426 +0000 UTC m=+581.167002437" watchObservedRunningTime="2026-09-16 18:53:20.906573486 +0000 UTC m=+581.168250497"
Sep 16 18:53:31.898136 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:53:31.898111    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-vrtsf"
Sep 16 18:55:22.758647 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.758612    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:55:22.762073 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.762054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.764614 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.764590    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:55:22.765256 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.765232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:55:22.765376 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.765296    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:55:22.765376 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.765299    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-v9sp9\""
Sep 16 18:55:22.765376 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.765368    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:55:22.765608 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.765593    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-v62w5\""
Sep 16 18:55:22.765682 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.765632    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:55:22.772752 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.772732    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:55:22.847731 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.847704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.847876 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.847742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjmrz\" (UniqueName: \"kubernetes.io/projected/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kube-api-access-fjmrz\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.847876 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.847780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.847876 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.847827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.847876 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.847853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.847876 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.847870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.848065 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.847905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd709db-d9ec-4f50-98ce-c7c1b7542344-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.848065 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.847939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.948600 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.948567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.948768 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.948603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.948768 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.948627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.948768 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.948738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd709db-d9ec-4f50-98ce-c7c1b7542344-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.948965 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.948792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.948965 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.948874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.948965 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.948941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjmrz\" (UniqueName: \"kubernetes.io/projected/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kube-api-access-fjmrz\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.949113 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.948993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.949113 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.949009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.949113 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.949054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.949498 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.949431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd709db-d9ec-4f50-98ce-c7c1b7542344-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.949498 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.949473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.949694 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.949503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.950995 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.950976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.951343 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.951330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.957155 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:22.957138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjmrz\" (UniqueName: \"kubernetes.io/projected/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kube-api-access-fjmrz\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:23.072466 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:23.072377    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:23.348970 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:23.348945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:55:23.351769 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:55:23.351726    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd709db_d9ec_4f50_98ce_c7c1b7542344.slice/crio-a0e275f3b4c87c1d7dd15c35b9944439bfe60bef45fdba418c7d754a658c949b WatchSource:0}: Error finding container a0e275f3b4c87c1d7dd15c35b9944439bfe60bef45fdba418c7d754a658c949b: Status 404 returned error can't find the container with id a0e275f3b4c87c1d7dd15c35b9944439bfe60bef45fdba418c7d754a658c949b
Sep 16 18:55:23.354310 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:23.354278    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:55:23.465428 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:23.465389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3cd709db-d9ec-4f50-98ce-c7c1b7542344","Type":"ContainerStarted","Data":"a0e275f3b4c87c1d7dd15c35b9944439bfe60bef45fdba418c7d754a658c949b"}
Sep 16 18:55:27.490325 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:27.490288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3cd709db-d9ec-4f50-98ce-c7c1b7542344","Type":"ContainerStarted","Data":"ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100"}
Sep 16 18:55:29.503126 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.503044    2580 generic.go:356] "Generic (PLEG): container finished" podID="3cd709db-d9ec-4f50-98ce-c7c1b7542344" containerID="ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100" exitCode=0
Sep 16 18:55:29.503556 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.503119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3cd709db-d9ec-4f50-98ce-c7c1b7542344","Type":"ContainerDied","Data":"ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100"}
Sep 16 18:55:29.504985 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.504958    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"]
Sep 16 18:55:29.513671 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.513627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:55:29.516970 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.516529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-x5zrc\""
Sep 16 18:55:29.516970 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.516795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:55:29.516970 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.516822    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-jpgd9\""
Sep 16 18:55:29.517237 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.517186    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:55:29.517688 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.517525    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:55:29.517688 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.517637    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"]
Sep 16 18:55:29.610056 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.610010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mg6s\" (UniqueName: \"kubernetes.io/projected/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-kube-api-access-9mg6s\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w\" (UID: \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:55:29.610229 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.610131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w\" (UID: \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:55:29.711197 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.711168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w\" (UID: \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:55:29.711367 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.711223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mg6s\" (UniqueName: \"kubernetes.io/projected/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-kube-api-access-9mg6s\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w\" (UID: \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:55:29.713916 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.713871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w\" (UID: \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:55:29.719075 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.719055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mg6s\" (UniqueName: \"kubernetes.io/projected/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-kube-api-access-9mg6s\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w\" (UID: \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:55:29.826858 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:29.826785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:55:30.107179 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:30.107098    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"]
Sep 16 18:55:30.114971 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:55:30.114940    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968e68f7_49e1_42a6_a0ad_bdfab6e79edc.slice/crio-8025e1a3e49527277f81c0b1311caf45efcd6dde148fed957f469a22659a9f10 WatchSource:0}: Error finding container 8025e1a3e49527277f81c0b1311caf45efcd6dde148fed957f469a22659a9f10: Status 404 returned error can't find the container with id 8025e1a3e49527277f81c0b1311caf45efcd6dde148fed957f469a22659a9f10
Sep 16 18:55:30.514862 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:30.514741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" event={"ID":"968e68f7-49e1-42a6-a0ad-bdfab6e79edc","Type":"ContainerStarted","Data":"8025e1a3e49527277f81c0b1311caf45efcd6dde148fed957f469a22659a9f10"}
Sep 16 18:55:34.551915 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:34.551856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" event={"ID":"968e68f7-49e1-42a6-a0ad-bdfab6e79edc","Type":"ContainerStarted","Data":"8d2b75d585fecba9ed5b0abcf39dac58aca09af7b611b4e5e51692585e315e83"}
Sep 16 18:55:34.552407 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:34.552004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:55:34.570805 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:55:34.570640    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" podStartSLOduration=1.765928019 podStartE2EDuration="5.570620974s" totalImagesPullingTime="3.804692955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:34.567939338 +0000 UTC m=+714.829616350" watchObservedRunningTime="2026-09-16 18:55:34.570620974 +0000 UTC m=+714.832297987"
Sep 16 18:56:00.708290 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:00.708256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3cd709db-d9ec-4f50-98ce-c7c1b7542344","Type":"ContainerStarted","Data":"0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a"}
Sep 16 18:56:00.728530 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:00.728483    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.4898757959999998 podStartE2EDuration="38.728468012s" totalImagesPullingTime="34.080106383s" totalInitContainerRuntime="2.158485833s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:00.727475447 +0000 UTC m=+740.989152472" watchObservedRunningTime="2026-09-16 18:56:00.728468012 +0000 UTC m=+740.990145024"
Sep 16 18:56:05.561217 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:05.561191    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:56:11.835775 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:11.835742    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"]
Sep 16 18:56:15.560640 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:15.560589    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:56:25.560429 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:25.560393    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:56:26.856125 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:26.856087    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerName="main" containerID="cri-o://8d2b75d585fecba9ed5b0abcf39dac58aca09af7b611b4e5e51692585e315e83" gracePeriod=45
Sep 16 18:56:35.560974 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:35.560930    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:56:35.561391 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:35.561041    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:56:45.561213 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:45.561175    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:56:55.560751 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:55.560707    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:56:56.981044 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:56.981012    2580 generic.go:356] "Generic (PLEG): container finished" podID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerID="8d2b75d585fecba9ed5b0abcf39dac58aca09af7b611b4e5e51692585e315e83" exitCode=0
Sep 16 18:56:56.981357 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:56.981090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" event={"ID":"968e68f7-49e1-42a6-a0ad-bdfab6e79edc","Type":"ContainerDied","Data":"8d2b75d585fecba9ed5b0abcf39dac58aca09af7b611b4e5e51692585e315e83"}
Sep 16 18:56:57.101707 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.101675    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:56:57.259380 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.259351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-tls-certs\") pod \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\" (UID: \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\") "
Sep 16 18:56:57.259549 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.259397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mg6s\" (UniqueName: \"kubernetes.io/projected/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-kube-api-access-9mg6s\") pod \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\" (UID: \"968e68f7-49e1-42a6-a0ad-bdfab6e79edc\") "
Sep 16 18:56:57.261672 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.261645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "968e68f7-49e1-42a6-a0ad-bdfab6e79edc" (UID: "968e68f7-49e1-42a6-a0ad-bdfab6e79edc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:57.261818 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.261728    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-kube-api-access-9mg6s" (OuterVolumeSpecName: "kube-api-access-9mg6s") pod "968e68f7-49e1-42a6-a0ad-bdfab6e79edc" (UID: "968e68f7-49e1-42a6-a0ad-bdfab6e79edc"). InnerVolumeSpecName "kube-api-access-9mg6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:57.361000 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.360969    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mg6s\" (UniqueName: \"kubernetes.io/projected/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-kube-api-access-9mg6s\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:56:57.361000 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.360994    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/968e68f7-49e1-42a6-a0ad-bdfab6e79edc-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:56:57.987617 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.987589    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"
Sep 16 18:56:57.988008 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.987614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w" event={"ID":"968e68f7-49e1-42a6-a0ad-bdfab6e79edc","Type":"ContainerDied","Data":"8025e1a3e49527277f81c0b1311caf45efcd6dde148fed957f469a22659a9f10"}
Sep 16 18:56:57.988008 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:57.987678    2580 scope.go:117] "RemoveContainer" containerID="8d2b75d585fecba9ed5b0abcf39dac58aca09af7b611b4e5e51692585e315e83"
Sep 16 18:56:58.008754 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:58.008725    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"]
Sep 16 18:56:58.014098 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:58.014074    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-k2h7w"]
Sep 16 18:56:58.277206 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:56:58.277123    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" path="/var/lib/kubelet/pods/968e68f7-49e1-42a6-a0ad-bdfab6e79edc/volumes"
Sep 16 18:57:06.207740 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.207639    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"]
Sep 16 18:57:06.208306 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.208052    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerName="main"
Sep 16 18:57:06.208306 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.208065    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerName="main"
Sep 16 18:57:06.208306 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.208132    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="968e68f7-49e1-42a6-a0ad-bdfab6e79edc" containerName="main"
Sep 16 18:57:06.210198 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.210178    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:06.213318 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.213297    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 18:57:06.213442 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.213301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 18:57:06.213442 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.213301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-8ks8c\""
Sep 16 18:57:06.213700 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.213685    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-c5ftv\""
Sep 16 18:57:06.224436 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.224416    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 18:57:06.243030 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.243007    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"]
Sep 16 18:57:06.351321 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.351293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92143404-eee3-4832-9b72-0b97c81774f3-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj\" (UID: \"92143404-eee3-4832-9b72-0b97c81774f3\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:06.351321 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.351324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5hv\" (UniqueName: \"kubernetes.io/projected/92143404-eee3-4832-9b72-0b97c81774f3-kube-api-access-nd5hv\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj\" (UID: \"92143404-eee3-4832-9b72-0b97c81774f3\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:06.452687 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.452654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92143404-eee3-4832-9b72-0b97c81774f3-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj\" (UID: \"92143404-eee3-4832-9b72-0b97c81774f3\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:06.452687 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.452688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5hv\" (UniqueName: \"kubernetes.io/projected/92143404-eee3-4832-9b72-0b97c81774f3-kube-api-access-nd5hv\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj\" (UID: \"92143404-eee3-4832-9b72-0b97c81774f3\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:06.455279 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.455250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92143404-eee3-4832-9b72-0b97c81774f3-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj\" (UID: \"92143404-eee3-4832-9b72-0b97c81774f3\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:06.471228 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.471172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5hv\" (UniqueName: \"kubernetes.io/projected/92143404-eee3-4832-9b72-0b97c81774f3-kube-api-access-nd5hv\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj\" (UID: \"92143404-eee3-4832-9b72-0b97c81774f3\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:06.523158 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.523132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:06.808456 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:06.808360    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"]
Sep 16 18:57:06.813482 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:57:06.813449    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92143404_eee3_4832_9b72_0b97c81774f3.slice/crio-2dec4f0afdffa00c326ed34093557811723998d3681592d088c8dea1e27db386 WatchSource:0}: Error finding container 2dec4f0afdffa00c326ed34093557811723998d3681592d088c8dea1e27db386: Status 404 returned error can't find the container with id 2dec4f0afdffa00c326ed34093557811723998d3681592d088c8dea1e27db386
Sep 16 18:57:07.037845 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:07.037789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" event={"ID":"92143404-eee3-4832-9b72-0b97c81774f3","Type":"ContainerStarted","Data":"a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255"}
Sep 16 18:57:07.038066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:07.037869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" event={"ID":"92143404-eee3-4832-9b72-0b97c81774f3","Type":"ContainerStarted","Data":"2dec4f0afdffa00c326ed34093557811723998d3681592d088c8dea1e27db386"}
Sep 16 18:57:07.038066 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:07.037982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:07.062224 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:07.062124    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" podStartSLOduration=1.06210619 podStartE2EDuration="1.06210619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:07.059678168 +0000 UTC m=+807.321355181" watchObservedRunningTime="2026-09-16 18:57:07.06210619 +0000 UTC m=+807.323783226"
Sep 16 18:57:38.043998 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:38.043967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:57:45.060099 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:45.060064    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"]
Sep 16 18:57:48.044061 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:48.044026    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" podUID="92143404-eee3-4832-9b72-0b97c81774f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:56.004135 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:56.004099    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:57:58.043873 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:57:58.043832    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" podUID="92143404-eee3-4832-9b72-0b97c81774f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:00.089723 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:00.089668    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" podUID="92143404-eee3-4832-9b72-0b97c81774f3" containerName="main" containerID="cri-o://a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255" gracePeriod=45
Sep 16 18:58:08.044556 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:08.044518    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" podUID="92143404-eee3-4832-9b72-0b97c81774f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:08.045023 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:08.044622    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:58:11.026580 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.026544    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="3cd709db-d9ec-4f50-98ce-c7c1b7542344" containerName="main" containerID="cri-o://0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a" gracePeriod=45
Sep 16 18:58:11.764679 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.764655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:11.839423 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839338    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd709db-d9ec-4f50-98ce-c7c1b7542344-cabundle-cert\") pod \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") "
Sep 16 18:58:11.839423 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kserve-provision-location\") pod \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") "
Sep 16 18:58:11.839636 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839424    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-dshm\") pod \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") "
Sep 16 18:58:11.839636 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839468    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tmp-dir\") pod \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") "
Sep 16 18:58:11.839636 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-model-cache\") pod \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") "
Sep 16 18:58:11.839636 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tls-certs\") pod \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") "
Sep 16 18:58:11.839636 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-home\") pod \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") "
Sep 16 18:58:11.839636 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjmrz\" (UniqueName: \"kubernetes.io/projected/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kube-api-access-fjmrz\") pod \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\" (UID: \"3cd709db-d9ec-4f50-98ce-c7c1b7542344\") "
Sep 16 18:58:11.839968 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd709db-d9ec-4f50-98ce-c7c1b7542344-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3cd709db-d9ec-4f50-98ce-c7c1b7542344" (UID: "3cd709db-d9ec-4f50-98ce-c7c1b7542344"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:11.839968 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839815    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-model-cache" (OuterVolumeSpecName: "model-cache") pod "3cd709db-d9ec-4f50-98ce-c7c1b7542344" (UID: "3cd709db-d9ec-4f50-98ce-c7c1b7542344"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:11.840085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839975    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd709db-d9ec-4f50-98ce-c7c1b7542344-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:11.840085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.839994    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:11.840620 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.840587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-home" (OuterVolumeSpecName: "home") pod "3cd709db-d9ec-4f50-98ce-c7c1b7542344" (UID: "3cd709db-d9ec-4f50-98ce-c7c1b7542344"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:11.842043 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.842015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-dshm" (OuterVolumeSpecName: "dshm") pod "3cd709db-d9ec-4f50-98ce-c7c1b7542344" (UID: "3cd709db-d9ec-4f50-98ce-c7c1b7542344"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:11.842152 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.842037    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3cd709db-d9ec-4f50-98ce-c7c1b7542344" (UID: "3cd709db-d9ec-4f50-98ce-c7c1b7542344"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:11.842195 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.842166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kube-api-access-fjmrz" (OuterVolumeSpecName: "kube-api-access-fjmrz") pod "3cd709db-d9ec-4f50-98ce-c7c1b7542344" (UID: "3cd709db-d9ec-4f50-98ce-c7c1b7542344"). InnerVolumeSpecName "kube-api-access-fjmrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:11.857466 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.857431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3cd709db-d9ec-4f50-98ce-c7c1b7542344" (UID: "3cd709db-d9ec-4f50-98ce-c7c1b7542344"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:11.866327 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.866306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3cd709db-d9ec-4f50-98ce-c7c1b7542344" (UID: "3cd709db-d9ec-4f50-98ce-c7c1b7542344"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:11.940671 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.940642    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:11.940671 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.940662    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd709db-d9ec-4f50-98ce-c7c1b7542344-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:11.940671 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.940672    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:11.940862 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.940681    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjmrz\" (UniqueName: \"kubernetes.io/projected/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kube-api-access-fjmrz\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:11.940862 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.940689    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:11.940862 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:11.940699    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd709db-d9ec-4f50-98ce-c7c1b7542344-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:12.360479 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.360439    2580 generic.go:356] "Generic (PLEG): container finished" podID="3cd709db-d9ec-4f50-98ce-c7c1b7542344" containerID="0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a" exitCode=0
Sep 16 18:58:12.360848 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.360493    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:12.360848 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.360518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3cd709db-d9ec-4f50-98ce-c7c1b7542344","Type":"ContainerDied","Data":"0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a"}
Sep 16 18:58:12.360848 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.360563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3cd709db-d9ec-4f50-98ce-c7c1b7542344","Type":"ContainerDied","Data":"a0e275f3b4c87c1d7dd15c35b9944439bfe60bef45fdba418c7d754a658c949b"}
Sep 16 18:58:12.360848 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.360585    2580 scope.go:117] "RemoveContainer" containerID="0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a"
Sep 16 18:58:12.369261 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.369230    2580 scope.go:117] "RemoveContainer" containerID="ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100"
Sep 16 18:58:12.376794 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.376779    2580 scope.go:117] "RemoveContainer" containerID="0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a"
Sep 16 18:58:12.377108 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:58:12.377090    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a\": container with ID starting with 0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a not found: ID does not exist" containerID="0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a"
Sep 16 18:58:12.377159 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.377121    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a"} err="failed to get container status \"0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a\": rpc error: code = NotFound desc = could not find container \"0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a\": container with ID starting with 0edf22ea0ca9df35e07319dc1319ba8187128ffb20d0865a2e183be035cd303a not found: ID does not exist"
Sep 16 18:58:12.377159 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.377137    2580 scope.go:117] "RemoveContainer" containerID="ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100"
Sep 16 18:58:12.377402 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:58:12.377382    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100\": container with ID starting with ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100 not found: ID does not exist" containerID="ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100"
Sep 16 18:58:12.377458 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.377408    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100"} err="failed to get container status \"ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100\": rpc error: code = NotFound desc = could not find container \"ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100\": container with ID starting with ca0f3b88cea42e8760559b689919292f7f9f309a7672284ae8f5dec33e8ca100 not found: ID does not exist"
Sep 16 18:58:12.387445 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.387419    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:58:12.393163 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:12.393143    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:58:14.277279 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:14.277245    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd709db-d9ec-4f50-98ce-c7c1b7542344" path="/var/lib/kubelet/pods/3cd709db-d9ec-4f50-98ce-c7c1b7542344/volumes"
Sep 16 18:58:18.044138 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:18.044100    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" podUID="92143404-eee3-4832-9b72-0b97c81774f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:28.043989 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:28.043951    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" podUID="92143404-eee3-4832-9b72-0b97c81774f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:30.331234 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.331209    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:58:30.450959 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.450927    2580 generic.go:356] "Generic (PLEG): container finished" podID="92143404-eee3-4832-9b72-0b97c81774f3" containerID="a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255" exitCode=0
Sep 16 18:58:30.451152 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.450968    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"
Sep 16 18:58:30.451152 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.451009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" event={"ID":"92143404-eee3-4832-9b72-0b97c81774f3","Type":"ContainerDied","Data":"a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255"}
Sep 16 18:58:30.451152 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.451062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj" event={"ID":"92143404-eee3-4832-9b72-0b97c81774f3","Type":"ContainerDied","Data":"2dec4f0afdffa00c326ed34093557811723998d3681592d088c8dea1e27db386"}
Sep 16 18:58:30.451152 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.451083    2580 scope.go:117] "RemoveContainer" containerID="a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255"
Sep 16 18:58:30.460597 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.460566    2580 scope.go:117] "RemoveContainer" containerID="a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255"
Sep 16 18:58:30.460965 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:58:30.460940    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255\": container with ID starting with a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255 not found: ID does not exist" containerID="a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255"
Sep 16 18:58:30.461053 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.460975    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255"} err="failed to get container status \"a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255\": rpc error: code = NotFound desc = could not find container \"a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255\": container with ID starting with a49191050a35cc4d76c7bf10556a90485f8afa1b09d2562cfd43604e0427f255 not found: ID does not exist"
Sep 16 18:58:30.503822 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.503801    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92143404-eee3-4832-9b72-0b97c81774f3-tls-certs\") pod \"92143404-eee3-4832-9b72-0b97c81774f3\" (UID: \"92143404-eee3-4832-9b72-0b97c81774f3\") "
Sep 16 18:58:30.503950 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.503829    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd5hv\" (UniqueName: \"kubernetes.io/projected/92143404-eee3-4832-9b72-0b97c81774f3-kube-api-access-nd5hv\") pod \"92143404-eee3-4832-9b72-0b97c81774f3\" (UID: \"92143404-eee3-4832-9b72-0b97c81774f3\") "
Sep 16 18:58:30.505935 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.505910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92143404-eee3-4832-9b72-0b97c81774f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92143404-eee3-4832-9b72-0b97c81774f3" (UID: "92143404-eee3-4832-9b72-0b97c81774f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:30.506017 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.505978    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92143404-eee3-4832-9b72-0b97c81774f3-kube-api-access-nd5hv" (OuterVolumeSpecName: "kube-api-access-nd5hv") pod "92143404-eee3-4832-9b72-0b97c81774f3" (UID: "92143404-eee3-4832-9b72-0b97c81774f3"). InnerVolumeSpecName "kube-api-access-nd5hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:30.605264 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.605239    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92143404-eee3-4832-9b72-0b97c81774f3-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:30.605264 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.605262    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd5hv\" (UniqueName: \"kubernetes.io/projected/92143404-eee3-4832-9b72-0b97c81774f3-kube-api-access-nd5hv\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:58:30.771938 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.771829    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"]
Sep 16 18:58:30.774793 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:30.774764    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-dfzqj"]
Sep 16 18:58:32.275515 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:32.275484    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92143404-eee3-4832-9b72-0b97c81774f3" path="/var/lib/kubelet/pods/92143404-eee3-4832-9b72-0b97c81774f3/volumes"
Sep 16 18:58:35.312557 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.312522    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"]
Sep 16 18:58:35.313057 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.313042    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd709db-d9ec-4f50-98ce-c7c1b7542344" containerName="main"
Sep 16 18:58:35.313115 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.313059    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd709db-d9ec-4f50-98ce-c7c1b7542344" containerName="main"
Sep 16 18:58:35.313115 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.313071    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92143404-eee3-4832-9b72-0b97c81774f3" containerName="main"
Sep 16 18:58:35.313115 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.313076    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92143404-eee3-4832-9b72-0b97c81774f3" containerName="main"
Sep 16 18:58:35.313115 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.313093    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd709db-d9ec-4f50-98ce-c7c1b7542344" containerName="storage-initializer"
Sep 16 18:58:35.313115 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.313102    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd709db-d9ec-4f50-98ce-c7c1b7542344" containerName="storage-initializer"
Sep 16 18:58:35.313285 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.313171    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd709db-d9ec-4f50-98ce-c7c1b7542344" containerName="main"
Sep 16 18:58:35.313285 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.313180    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92143404-eee3-4832-9b72-0b97c81774f3" containerName="main"
Sep 16 18:58:35.316072 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.316051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.318566 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.318543    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 18:58:35.318566 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.318562    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 18:58:35.318742 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.318562    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:58:35.318958 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.318943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-2zzk5\""
Sep 16 18:58:35.328697 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.328668    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"]
Sep 16 18:58:35.380261 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.380235    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"]
Sep 16 18:58:35.383473 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.383457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:58:35.385693 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.385670    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 18:58:35.385693 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.385691    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 18:58:35.385861 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.385709    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-fpp89\""
Sep 16 18:58:35.385861 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.385691    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 18:58:35.385989 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.385868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-l9hdm\""
Sep 16 18:58:35.393250 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.393230    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"]
Sep 16 18:58:35.445642 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.445611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.445777 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.445656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.445777 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.445680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppp5w\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-kube-api-access-ppp5w\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.445777 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.445703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.445777 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.445717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8403589b-a06e-488a-bbee-0368ec3868a6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.445777 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.445753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.445983 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.445826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8403589b-a06e-488a-bbee-0368ec3868a6-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.445983 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.445847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.445983 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.445860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.546717 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.546684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.546924 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.546730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.546924 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.546843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp5w\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-kube-api-access-ppp5w\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547054 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.546918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rv8q\" (UniqueName: \"kubernetes.io/projected/86841c86-177c-496b-8bac-e3e668d21972-kube-api-access-6rv8q\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js\" (UID: \"86841c86-177c-496b-8bac-e3e668d21972\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:58:35.547054 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.546960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547054 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.546986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8403589b-a06e-488a-bbee-0368ec3868a6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547194 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547194 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547194 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8403589b-a06e-488a-bbee-0368ec3868a6-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547194 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86841c86-177c-496b-8bac-e3e668d21972-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js\" (UID: \"86841c86-177c-496b-8bac-e3e668d21972\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:58:35.547387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547387 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547595 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8403589b-a06e-488a-bbee-0368ec3868a6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.547645 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.547591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.549511 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.549486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.549706 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.549688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8403589b-a06e-488a-bbee-0368ec3868a6-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.554266 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.554241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.554465 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.554451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppp5w\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-kube-api-access-ppp5w\") pod \"router-gateway-1-openshift-default-5b4d9458bd-9khr4\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.629451 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.629380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:35.647753 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.647728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86841c86-177c-496b-8bac-e3e668d21972-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js\" (UID: \"86841c86-177c-496b-8bac-e3e668d21972\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:58:35.647846 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.647785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rv8q\" (UniqueName: \"kubernetes.io/projected/86841c86-177c-496b-8bac-e3e668d21972-kube-api-access-6rv8q\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js\" (UID: \"86841c86-177c-496b-8bac-e3e668d21972\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:58:35.650374 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.650349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86841c86-177c-496b-8bac-e3e668d21972-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js\" (UID: \"86841c86-177c-496b-8bac-e3e668d21972\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:58:35.657169 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.657139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rv8q\" (UniqueName: \"kubernetes.io/projected/86841c86-177c-496b-8bac-e3e668d21972-kube-api-access-6rv8q\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js\" (UID: \"86841c86-177c-496b-8bac-e3e668d21972\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:58:35.694793 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.694498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:58:35.906076 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.906050    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"]
Sep 16 18:58:35.908685 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:58:35.908659    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8403589b_a06e_488a_bbee_0368ec3868a6.slice/crio-d7d538b072f40620e22933c1bc346c5d5097b3767111b358c07cd3333577912f WatchSource:0}: Error finding container d7d538b072f40620e22933c1bc346c5d5097b3767111b358c07cd3333577912f: Status 404 returned error can't find the container with id d7d538b072f40620e22933c1bc346c5d5097b3767111b358c07cd3333577912f
Sep 16 18:58:35.966720 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:35.966693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"]
Sep 16 18:58:35.969514 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:58:35.969490    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86841c86_177c_496b_8bac_e3e668d21972.slice/crio-5f1863d5a1e3f1c0cfa456116fbe963f930305d50be2b8de2495d5cdd9cc706d WatchSource:0}: Error finding container 5f1863d5a1e3f1c0cfa456116fbe963f930305d50be2b8de2495d5cdd9cc706d: Status 404 returned error can't find the container with id 5f1863d5a1e3f1c0cfa456116fbe963f930305d50be2b8de2495d5cdd9cc706d
Sep 16 18:58:36.483979 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:36.483937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" event={"ID":"86841c86-177c-496b-8bac-e3e668d21972","Type":"ContainerStarted","Data":"47a22d4367fcfcf9f623c7d78c07e9ae09f84ee604d7d6186e6bc0beb437f4cd"}
Sep 16 18:58:36.484424 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:36.484077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:58:36.484424 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:36.484113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" event={"ID":"86841c86-177c-496b-8bac-e3e668d21972","Type":"ContainerStarted","Data":"5f1863d5a1e3f1c0cfa456116fbe963f930305d50be2b8de2495d5cdd9cc706d"}
Sep 16 18:58:36.485345 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:36.485321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4" event={"ID":"8403589b-a06e-488a-bbee-0368ec3868a6","Type":"ContainerStarted","Data":"d7d538b072f40620e22933c1bc346c5d5097b3767111b358c07cd3333577912f"}
Sep 16 18:58:36.501947 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:36.501884    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" podStartSLOduration=1.501869282 podStartE2EDuration="1.501869282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:36.501651297 +0000 UTC m=+896.763328390" watchObservedRunningTime="2026-09-16 18:58:36.501869282 +0000 UTC m=+896.763546294"
Sep 16 18:58:38.766546 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:38.766514    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:58:38.766767 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:38.766586    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:58:38.766767 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:38.766627    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:58:39.503884 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:39.503853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4" event={"ID":"8403589b-a06e-488a-bbee-0368ec3868a6","Type":"ContainerStarted","Data":"7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5"}
Sep 16 18:58:39.524960 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:39.524911    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4" podStartSLOduration=1.669278039 podStartE2EDuration="4.524878078s" totalImagesPullingTime="2.855600039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:39.523798446 +0000 UTC m=+899.785475460" watchObservedRunningTime="2026-09-16 18:58:39.524878078 +0000 UTC m=+899.786555090"
Sep 16 18:58:39.629714 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:39.629687    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:39.642009 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:39.641971    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:40.508885 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:40.508806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:58:40.509757 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:58:40.509741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:59:07.493320 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:07.493288    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 18:59:23.762053 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.762022    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"]
Sep 16 18:59:23.780400 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.779129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"]
Sep 16 18:59:23.780400 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.779291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.783015 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.782987    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 18:59:23.783477 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.783412    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:59:23.783618 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.783594    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-575zn\""
Sep 16 18:59:23.783836 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.783625    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 18:59:23.860553 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.860477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.860761 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.860578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.860761 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.860630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.860761 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.860657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3512d24-3c01-4cba-aedd-6e97711e524b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.860761 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.860682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mnc\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-kube-api-access-j8mnc\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.860761 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.860738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.861085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.860783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.861085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.860834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.861085 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.860861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.962384 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.962333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.962573 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.962405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.962573 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.962456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.962573 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.962504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3512d24-3c01-4cba-aedd-6e97711e524b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.962573 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.962532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mnc\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-kube-api-access-j8mnc\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.962806 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.962588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.962806 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.962630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.962806 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.962684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.962806 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.962715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.963598 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.963555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.964679 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.964251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.964679 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.964283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.964979 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.964951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3512d24-3c01-4cba-aedd-6e97711e524b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.965559 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.965538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.967567 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.967546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.967714 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.967601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.975846 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.975554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mnc\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-kube-api-access-j8mnc\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:23.975846 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:23.975808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-qdsv6\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:24.091792 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:24.091711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:24.372827 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:24.372737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"]
Sep 16 18:59:24.376964 ip-10-0-139-95 kubenswrapper[2580]: W0916 18:59:24.376935    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3512d24_3c01_4cba_aedd_6e97711e524b.slice/crio-c2d344cbdb0ca5560549724f3cfb2d0bed61657e66954756ebbaf9a180aa99a0 WatchSource:0}: Error finding container c2d344cbdb0ca5560549724f3cfb2d0bed61657e66954756ebbaf9a180aa99a0: Status 404 returned error can't find the container with id c2d344cbdb0ca5560549724f3cfb2d0bed61657e66954756ebbaf9a180aa99a0
Sep 16 18:59:24.379042 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:24.379011    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:59:24.379123 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:24.379077    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:59:24.379123 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:24.379103    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 18:59:24.732729 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:24.732693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6" event={"ID":"d3512d24-3c01-4cba-aedd-6e97711e524b","Type":"ContainerStarted","Data":"f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea"}
Sep 16 18:59:24.732926 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:24.732773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6" event={"ID":"d3512d24-3c01-4cba-aedd-6e97711e524b","Type":"ContainerStarted","Data":"c2d344cbdb0ca5560549724f3cfb2d0bed61657e66954756ebbaf9a180aa99a0"}
Sep 16 18:59:24.813816 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:24.813758    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6" podStartSLOduration=1.813742145 podStartE2EDuration="1.813742145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:24.811102345 +0000 UTC m=+945.072779358" watchObservedRunningTime="2026-09-16 18:59:24.813742145 +0000 UTC m=+945.075419182"
Sep 16 18:59:25.092807 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:25.092724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:25.097360 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:25.097337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:25.737326 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:25.737297    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:25.738272 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:25.738251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:28.418677 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:28.418643    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"]
Sep 16 18:59:28.419126 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:28.418880    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4" podUID="8403589b-a06e-488a-bbee-0368ec3868a6" containerName="istio-proxy" containerID="cri-o://7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5" gracePeriod=30
Sep 16 18:59:31.371284 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.371259    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:59:31.433530 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433507    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-data\") pod \"8403589b-a06e-488a-bbee-0368ec3868a6\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") "
Sep 16 18:59:31.433664 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppp5w\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-kube-api-access-ppp5w\") pod \"8403589b-a06e-488a-bbee-0368ec3868a6\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") "
Sep 16 18:59:31.433664 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-istio-token\") pod \"8403589b-a06e-488a-bbee-0368ec3868a6\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") "
Sep 16 18:59:31.433664 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433634    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8403589b-a06e-488a-bbee-0368ec3868a6-istiod-ca-cert\") pod \"8403589b-a06e-488a-bbee-0368ec3868a6\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") "
Sep 16 18:59:31.433664 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-certs\") pod \"8403589b-a06e-488a-bbee-0368ec3868a6\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") "
Sep 16 18:59:31.433871 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-credential-socket\") pod \"8403589b-a06e-488a-bbee-0368ec3868a6\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") "
Sep 16 18:59:31.433871 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-socket\") pod \"8403589b-a06e-488a-bbee-0368ec3868a6\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") "
Sep 16 18:59:31.433871 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8403589b-a06e-488a-bbee-0368ec3868a6-istio-podinfo\") pod \"8403589b-a06e-488a-bbee-0368ec3868a6\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") "
Sep 16 18:59:31.433871 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433811    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-envoy\") pod \"8403589b-a06e-488a-bbee-0368ec3868a6\" (UID: \"8403589b-a06e-488a-bbee-0368ec3868a6\") "
Sep 16 18:59:31.433871 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.433824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "8403589b-a06e-488a-bbee-0368ec3868a6" (UID: "8403589b-a06e-488a-bbee-0368ec3868a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:31.434244 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.434093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8403589b-a06e-488a-bbee-0368ec3868a6" (UID: "8403589b-a06e-488a-bbee-0368ec3868a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:31.434244 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.434123    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-data\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:31.434350 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.434283    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8403589b-a06e-488a-bbee-0368ec3868a6" (UID: "8403589b-a06e-488a-bbee-0368ec3868a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:31.434350 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.434330    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8403589b-a06e-488a-bbee-0368ec3868a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8403589b-a06e-488a-bbee-0368ec3868a6" (UID: "8403589b-a06e-488a-bbee-0368ec3868a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:31.434446 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.434394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8403589b-a06e-488a-bbee-0368ec3868a6" (UID: "8403589b-a06e-488a-bbee-0368ec3868a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:31.436918 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.436833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8403589b-a06e-488a-bbee-0368ec3868a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8403589b-a06e-488a-bbee-0368ec3868a6" (UID: "8403589b-a06e-488a-bbee-0368ec3868a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:59:31.436918 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.436840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8403589b-a06e-488a-bbee-0368ec3868a6" (UID: "8403589b-a06e-488a-bbee-0368ec3868a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:31.437071 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.436930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-kube-api-access-ppp5w" (OuterVolumeSpecName: "kube-api-access-ppp5w") pod "8403589b-a06e-488a-bbee-0368ec3868a6" (UID: "8403589b-a06e-488a-bbee-0368ec3868a6"). InnerVolumeSpecName "kube-api-access-ppp5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:31.437071 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.436940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "8403589b-a06e-488a-bbee-0368ec3868a6" (UID: "8403589b-a06e-488a-bbee-0368ec3868a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:31.534799 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.534766    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8403589b-a06e-488a-bbee-0368ec3868a6-istiod-ca-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:31.534799 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.534792    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:31.535019 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.534806    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-credential-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:31.535019 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.534819    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-workload-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:31.535019 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.534831    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8403589b-a06e-488a-bbee-0368ec3868a6-istio-podinfo\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:31.535019 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.534842    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8403589b-a06e-488a-bbee-0368ec3868a6-istio-envoy\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:31.535019 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.534853    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppp5w\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-kube-api-access-ppp5w\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:31.535019 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.534864    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8403589b-a06e-488a-bbee-0368ec3868a6-istio-token\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:31.769061 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.768962    2580 generic.go:356] "Generic (PLEG): container finished" podID="8403589b-a06e-488a-bbee-0368ec3868a6" containerID="7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5" exitCode=0
Sep 16 18:59:31.769061 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.769008    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"
Sep 16 18:59:31.769061 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.769044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4" event={"ID":"8403589b-a06e-488a-bbee-0368ec3868a6","Type":"ContainerDied","Data":"7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5"}
Sep 16 18:59:31.769328 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.769096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4" event={"ID":"8403589b-a06e-488a-bbee-0368ec3868a6","Type":"ContainerDied","Data":"d7d538b072f40620e22933c1bc346c5d5097b3767111b358c07cd3333577912f"}
Sep 16 18:59:31.769328 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.769122    2580 scope.go:117] "RemoveContainer" containerID="7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5"
Sep 16 18:59:31.778311 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.778296    2580 scope.go:117] "RemoveContainer" containerID="7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5"
Sep 16 18:59:31.778601 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:59:31.778580    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5\": container with ID starting with 7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5 not found: ID does not exist" containerID="7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5"
Sep 16 18:59:31.778645 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.778611    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5"} err="failed to get container status \"7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5\": rpc error: code = NotFound desc = could not find container \"7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5\": container with ID starting with 7d0d6a6774e9d75c69742e83c84a7b0a1e37eafca97562d93ff26c8a2d589ab5 not found: ID does not exist"
Sep 16 18:59:31.789856 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.789828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"]
Sep 16 18:59:31.795937 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:31.795917    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-9khr4"]
Sep 16 18:59:32.276435 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:32.276391    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8403589b-a06e-488a-bbee-0368ec3868a6" path="/var/lib/kubelet/pods/8403589b-a06e-488a-bbee-0368ec3868a6/volumes"
Sep 16 18:59:45.865545 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:45.865502    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"]
Sep 16 18:59:47.493852 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:47.493816    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" podUID="86841c86-177c-496b-8bac-e3e668d21972" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:50.817988 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:50.817948    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"]
Sep 16 18:59:50.818420 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:50.818164    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6" podUID="d3512d24-3c01-4cba-aedd-6e97711e524b" containerName="istio-proxy" containerID="cri-o://f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea" gracePeriod=30
Sep 16 18:59:53.260242 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.260216    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:53.332129 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332103    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-credential-socket\") pod \"d3512d24-3c01-4cba-aedd-6e97711e524b\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") "
Sep 16 18:59:53.332268 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332159    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-token\") pod \"d3512d24-3c01-4cba-aedd-6e97711e524b\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") "
Sep 16 18:59:53.332268 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-data\") pod \"d3512d24-3c01-4cba-aedd-6e97711e524b\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") "
Sep 16 18:59:53.332268 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-envoy\") pod \"d3512d24-3c01-4cba-aedd-6e97711e524b\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") "
Sep 16 18:59:53.332268 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332266    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3512d24-3c01-4cba-aedd-6e97711e524b-istiod-ca-cert\") pod \"d3512d24-3c01-4cba-aedd-6e97711e524b\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") "
Sep 16 18:59:53.332451 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-podinfo\") pod \"d3512d24-3c01-4cba-aedd-6e97711e524b\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") "
Sep 16 18:59:53.332451 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-certs\") pod \"d3512d24-3c01-4cba-aedd-6e97711e524b\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") "
Sep 16 18:59:53.332451 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8mnc\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-kube-api-access-j8mnc\") pod \"d3512d24-3c01-4cba-aedd-6e97711e524b\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") "
Sep 16 18:59:53.332451 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-socket\") pod \"d3512d24-3c01-4cba-aedd-6e97711e524b\" (UID: \"d3512d24-3c01-4cba-aedd-6e97711e524b\") "
Sep 16 18:59:53.332451 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3512d24-3c01-4cba-aedd-6e97711e524b" (UID: "d3512d24-3c01-4cba-aedd-6e97711e524b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:53.332705 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3512d24-3c01-4cba-aedd-6e97711e524b" (UID: "d3512d24-3c01-4cba-aedd-6e97711e524b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:53.332705 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3512d24-3c01-4cba-aedd-6e97711e524b" (UID: "d3512d24-3c01-4cba-aedd-6e97711e524b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:53.332808 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3512d24-3c01-4cba-aedd-6e97711e524b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3512d24-3c01-4cba-aedd-6e97711e524b" (UID: "d3512d24-3c01-4cba-aedd-6e97711e524b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:53.332921 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3512d24-3c01-4cba-aedd-6e97711e524b" (UID: "d3512d24-3c01-4cba-aedd-6e97711e524b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:53.333013 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.332995    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-data\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.333074 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.333016    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3512d24-3c01-4cba-aedd-6e97711e524b-istiod-ca-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.333074 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.333031    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.333074 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.333043    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-workload-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.333074 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.333056    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-credential-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.335113 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.335085    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3512d24-3c01-4cba-aedd-6e97711e524b" (UID: "d3512d24-3c01-4cba-aedd-6e97711e524b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:53.335232 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.335118    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3512d24-3c01-4cba-aedd-6e97711e524b" (UID: "d3512d24-3c01-4cba-aedd-6e97711e524b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:53.335232 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.335181    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3512d24-3c01-4cba-aedd-6e97711e524b" (UID: "d3512d24-3c01-4cba-aedd-6e97711e524b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:59:53.335232 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.335191    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-kube-api-access-j8mnc" (OuterVolumeSpecName: "kube-api-access-j8mnc") pod "d3512d24-3c01-4cba-aedd-6e97711e524b" (UID: "d3512d24-3c01-4cba-aedd-6e97711e524b"). InnerVolumeSpecName "kube-api-access-j8mnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:53.434267 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.434243    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-token\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.434267 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.434265    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-envoy\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.434442 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.434275    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3512d24-3c01-4cba-aedd-6e97711e524b-istio-podinfo\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.434442 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.434283    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8mnc\" (UniqueName: \"kubernetes.io/projected/d3512d24-3c01-4cba-aedd-6e97711e524b-kube-api-access-j8mnc\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.881475 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.881382    2580 generic.go:356] "Generic (PLEG): container finished" podID="d3512d24-3c01-4cba-aedd-6e97711e524b" containerID="f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea" exitCode=0
Sep 16 18:59:53.881475 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.881446    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"
Sep 16 18:59:53.881678 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.881471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6" event={"ID":"d3512d24-3c01-4cba-aedd-6e97711e524b","Type":"ContainerDied","Data":"f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea"}
Sep 16 18:59:53.881678 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.881556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6" event={"ID":"d3512d24-3c01-4cba-aedd-6e97711e524b","Type":"ContainerDied","Data":"c2d344cbdb0ca5560549724f3cfb2d0bed61657e66954756ebbaf9a180aa99a0"}
Sep 16 18:59:53.881678 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.881578    2580 scope.go:117] "RemoveContainer" containerID="f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea"
Sep 16 18:59:53.892306 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.892274    2580 scope.go:117] "RemoveContainer" containerID="f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea"
Sep 16 18:59:53.892724 ip-10-0-139-95 kubenswrapper[2580]: E0916 18:59:53.892701    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea\": container with ID starting with f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea not found: ID does not exist" containerID="f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea"
Sep 16 18:59:53.892795 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.892737    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea"} err="failed to get container status \"f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea\": rpc error: code = NotFound desc = could not find container \"f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea\": container with ID starting with f112fae28fcbd94e6e2b6b995431c21aff3c8f420a2033fd3e0d408d01ac99ea not found: ID does not exist"
Sep 16 18:59:53.905032 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.905009    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"]
Sep 16 18:59:53.908516 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:53.908497    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-qdsv6"]
Sep 16 18:59:54.276481 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:54.276445    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3512d24-3c01-4cba-aedd-6e97711e524b" path="/var/lib/kubelet/pods/d3512d24-3c01-4cba-aedd-6e97711e524b/volumes"
Sep 16 18:59:57.493232 ip-10-0-139-95 kubenswrapper[2580]: I0916 18:59:57.493195    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" podUID="86841c86-177c-496b-8bac-e3e668d21972" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:00.899446 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:00.899351    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" podUID="86841c86-177c-496b-8bac-e3e668d21972" containerName="main" containerID="cri-o://47a22d4367fcfcf9f623c7d78c07e9ae09f84ee604d7d6186e6bc0beb437f4cd" gracePeriod=45
Sep 16 19:00:07.492796 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:07.492759    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" podUID="86841c86-177c-496b-8bac-e3e668d21972" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:07.493229 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:07.492944    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 19:00:17.493361 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:17.493314    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" podUID="86841c86-177c-496b-8bac-e3e668d21972" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:27.493549 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:27.493514    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" podUID="86841c86-177c-496b-8bac-e3e668d21972" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:31.060263 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:31.060230    2580 generic.go:356] "Generic (PLEG): container finished" podID="86841c86-177c-496b-8bac-e3e668d21972" containerID="47a22d4367fcfcf9f623c7d78c07e9ae09f84ee604d7d6186e6bc0beb437f4cd" exitCode=0
Sep 16 19:00:31.061039 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:31.060296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" event={"ID":"86841c86-177c-496b-8bac-e3e668d21972","Type":"ContainerDied","Data":"47a22d4367fcfcf9f623c7d78c07e9ae09f84ee604d7d6186e6bc0beb437f4cd"}
Sep 16 19:00:31.143571 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:31.143549    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 19:00:31.143847 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:31.143831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86841c86-177c-496b-8bac-e3e668d21972-tls-certs\") pod \"86841c86-177c-496b-8bac-e3e668d21972\" (UID: \"86841c86-177c-496b-8bac-e3e668d21972\") "
Sep 16 19:00:31.143905 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:31.143868    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rv8q\" (UniqueName: \"kubernetes.io/projected/86841c86-177c-496b-8bac-e3e668d21972-kube-api-access-6rv8q\") pod \"86841c86-177c-496b-8bac-e3e668d21972\" (UID: \"86841c86-177c-496b-8bac-e3e668d21972\") "
Sep 16 19:00:31.146145 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:31.146114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86841c86-177c-496b-8bac-e3e668d21972-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86841c86-177c-496b-8bac-e3e668d21972" (UID: "86841c86-177c-496b-8bac-e3e668d21972"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:31.146250 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:31.146211    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86841c86-177c-496b-8bac-e3e668d21972-kube-api-access-6rv8q" (OuterVolumeSpecName: "kube-api-access-6rv8q") pod "86841c86-177c-496b-8bac-e3e668d21972" (UID: "86841c86-177c-496b-8bac-e3e668d21972"). InnerVolumeSpecName "kube-api-access-6rv8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:31.245009 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:31.244983    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86841c86-177c-496b-8bac-e3e668d21972-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:00:31.245009 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:31.245006    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rv8q\" (UniqueName: \"kubernetes.io/projected/86841c86-177c-496b-8bac-e3e668d21972-kube-api-access-6rv8q\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:00:32.066559 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:32.066527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js" event={"ID":"86841c86-177c-496b-8bac-e3e668d21972","Type":"ContainerDied","Data":"5f1863d5a1e3f1c0cfa456116fbe963f930305d50be2b8de2495d5cdd9cc706d"}
Sep 16 19:00:32.066559 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:32.066551    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"
Sep 16 19:00:32.067063 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:32.066583    2580 scope.go:117] "RemoveContainer" containerID="47a22d4367fcfcf9f623c7d78c07e9ae09f84ee604d7d6186e6bc0beb437f4cd"
Sep 16 19:00:32.089168 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:32.089143    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"]
Sep 16 19:00:32.092771 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:32.092747    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcc9js"]
Sep 16 19:00:32.275876 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:32.275851    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86841c86-177c-496b-8bac-e3e668d21972" path="/var/lib/kubelet/pods/86841c86-177c-496b-8bac-e3e668d21972/volumes"
Sep 16 19:00:48.328564 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.328526    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"]
Sep 16 19:00:48.328977 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.328940    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86841c86-177c-496b-8bac-e3e668d21972" containerName="main"
Sep 16 19:00:48.328977 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.328953    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86841c86-177c-496b-8bac-e3e668d21972" containerName="main"
Sep 16 19:00:48.328977 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.328972    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3512d24-3c01-4cba-aedd-6e97711e524b" containerName="istio-proxy"
Sep 16 19:00:48.328977 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.328977    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3512d24-3c01-4cba-aedd-6e97711e524b" containerName="istio-proxy"
Sep 16 19:00:48.329103 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.328995    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8403589b-a06e-488a-bbee-0368ec3868a6" containerName="istio-proxy"
Sep 16 19:00:48.329103 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.329003    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8403589b-a06e-488a-bbee-0368ec3868a6" containerName="istio-proxy"
Sep 16 19:00:48.329103 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.329057    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86841c86-177c-496b-8bac-e3e668d21972" containerName="main"
Sep 16 19:00:48.329103 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.329068    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8403589b-a06e-488a-bbee-0368ec3868a6" containerName="istio-proxy"
Sep 16 19:00:48.329103 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.329079    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3512d24-3c01-4cba-aedd-6e97711e524b" containerName="istio-proxy"
Sep 16 19:00:48.333214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.333192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.335792 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.335770    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 19:00:48.336554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.336534    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-hfdsq\""
Sep 16 19:00:48.336554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.336561    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 19:00:48.336708 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.336540    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 19:00:48.345096 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.345068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"]
Sep 16 19:00:48.378095 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.378051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c57138-56c1-4545-8369-d5a726afbdf5-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.378287 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.378102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.378287 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.378178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.378287 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.378241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.378287 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.378273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24lk\" (UniqueName: \"kubernetes.io/projected/47c57138-56c1-4545-8369-d5a726afbdf5-kube-api-access-v24lk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.378426 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.378350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.479172 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.479111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.479451 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.479201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c57138-56c1-4545-8369-d5a726afbdf5-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.479451 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.479232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.479451 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.479270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.479451 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.479309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.479451 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.479355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24lk\" (UniqueName: \"kubernetes.io/projected/47c57138-56c1-4545-8369-d5a726afbdf5-kube-api-access-v24lk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.479731 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.479629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.479731 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.479664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.479731 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.479688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.481958 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.481931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.482223 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.482204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c57138-56c1-4545-8369-d5a726afbdf5-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.488615 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.488588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24lk\" (UniqueName: \"kubernetes.io/projected/47c57138-56c1-4545-8369-d5a726afbdf5-kube-api-access-v24lk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.644378 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.644273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:48.921916 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.921863    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"]
Sep 16 19:00:48.925289 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:00:48.925253    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c57138_56c1_4545_8369_d5a726afbdf5.slice/crio-af36c08bf4a5b450479c3f2eedd5a87d5487275fe602a661c0251d85dcdc5700 WatchSource:0}: Error finding container af36c08bf4a5b450479c3f2eedd5a87d5487275fe602a661c0251d85dcdc5700: Status 404 returned error can't find the container with id af36c08bf4a5b450479c3f2eedd5a87d5487275fe602a661c0251d85dcdc5700
Sep 16 19:00:48.927767 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:48.927746    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:00:49.144920 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:49.144860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l" event={"ID":"47c57138-56c1-4545-8369-d5a726afbdf5","Type":"ContainerStarted","Data":"af36c08bf4a5b450479c3f2eedd5a87d5487275fe602a661c0251d85dcdc5700"}
Sep 16 19:00:51.157117 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:51.157081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l" event={"ID":"47c57138-56c1-4545-8369-d5a726afbdf5","Type":"ContainerStarted","Data":"a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d"}
Sep 16 19:00:51.176546 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:51.176496    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l" podStartSLOduration=1.5323891870000002 podStartE2EDuration="3.176480805s" totalImagesPullingTime="1.644091618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:51.175502496 +0000 UTC m=+1031.437179507" watchObservedRunningTime="2026-09-16 19:00:51.176480805 +0000 UTC m=+1031.438157816"
Sep 16 19:00:51.644716 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:51.644682    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:58.645337 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:58.645306    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:58.657520 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:58.657499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:00:59.203779 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:00:59.203751    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:01:22.198675 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.198643    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"]
Sep 16 19:01:22.209329 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:01:22.209293    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:01:22.209329 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:22.209329 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:22.209329 ip-10-0-139-95 kubenswrapper[2580]:  > containerID="a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d" cmd=["/bin/sleep","15"]
Sep 16 19:01:22.209617 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.209352    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l" message="" err=<
Sep 16 19:01:22.209617 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:22.209617 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:22.209617 ip-10-0-139-95 kubenswrapper[2580]:  >
Sep 16 19:01:22.209617 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:01:22.209364    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:01:22.209617 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:22.209617 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:22.209617 ip-10-0-139-95 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l" podUID="47c57138-56c1-4545-8369-d5a726afbdf5" containerName="main" containerID="cri-o://a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d"
Sep 16 19:01:22.209617 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.209410    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l" podUID="47c57138-56c1-4545-8369-d5a726afbdf5" containerName="main" containerID="cri-o://a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d" gracePeriod=60
Sep 16 19:01:22.557249 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.557225    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:01:22.668914 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.668864    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-tmp-dir\") pod \"47c57138-56c1-4545-8369-d5a726afbdf5\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") "
Sep 16 19:01:22.669076 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.668931    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-dshm\") pod \"47c57138-56c1-4545-8369-d5a726afbdf5\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") "
Sep 16 19:01:22.669076 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.668955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v24lk\" (UniqueName: \"kubernetes.io/projected/47c57138-56c1-4545-8369-d5a726afbdf5-kube-api-access-v24lk\") pod \"47c57138-56c1-4545-8369-d5a726afbdf5\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") "
Sep 16 19:01:22.669076 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.668983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-model-cache\") pod \"47c57138-56c1-4545-8369-d5a726afbdf5\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") "
Sep 16 19:01:22.669076 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.669046    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c57138-56c1-4545-8369-d5a726afbdf5-tls-certs\") pod \"47c57138-56c1-4545-8369-d5a726afbdf5\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") "
Sep 16 19:01:22.669302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.669107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-home\") pod \"47c57138-56c1-4545-8369-d5a726afbdf5\" (UID: \"47c57138-56c1-4545-8369-d5a726afbdf5\") "
Sep 16 19:01:22.669302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.669136    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47c57138-56c1-4545-8369-d5a726afbdf5" (UID: "47c57138-56c1-4545-8369-d5a726afbdf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:22.669302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.669281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "47c57138-56c1-4545-8369-d5a726afbdf5" (UID: "47c57138-56c1-4545-8369-d5a726afbdf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:22.669451 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.669383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-home" (OuterVolumeSpecName: "home") pod "47c57138-56c1-4545-8369-d5a726afbdf5" (UID: "47c57138-56c1-4545-8369-d5a726afbdf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:22.669511 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.669467    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:01:22.669511 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.669486    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:01:22.671374 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.671354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c57138-56c1-4545-8369-d5a726afbdf5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47c57138-56c1-4545-8369-d5a726afbdf5" (UID: "47c57138-56c1-4545-8369-d5a726afbdf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:22.671374 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.671365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c57138-56c1-4545-8369-d5a726afbdf5-kube-api-access-v24lk" (OuterVolumeSpecName: "kube-api-access-v24lk") pod "47c57138-56c1-4545-8369-d5a726afbdf5" (UID: "47c57138-56c1-4545-8369-d5a726afbdf5"). InnerVolumeSpecName "kube-api-access-v24lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:22.671478 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.671425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-dshm" (OuterVolumeSpecName: "dshm") pod "47c57138-56c1-4545-8369-d5a726afbdf5" (UID: "47c57138-56c1-4545-8369-d5a726afbdf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:22.770390 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.770315    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:01:22.770390 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.770339    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c57138-56c1-4545-8369-d5a726afbdf5-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:01:22.770390 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.770349    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v24lk\" (UniqueName: \"kubernetes.io/projected/47c57138-56c1-4545-8369-d5a726afbdf5-kube-api-access-v24lk\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:01:22.770390 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:22.770360    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c57138-56c1-4545-8369-d5a726afbdf5-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:01:23.304287 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:23.304254    2580 generic.go:356] "Generic (PLEG): container finished" podID="47c57138-56c1-4545-8369-d5a726afbdf5" containerID="a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d" exitCode=0
Sep 16 19:01:23.304686 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:23.304309    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"
Sep 16 19:01:23.304686 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:23.304343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l" event={"ID":"47c57138-56c1-4545-8369-d5a726afbdf5","Type":"ContainerDied","Data":"a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d"}
Sep 16 19:01:23.304686 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:23.304403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l" event={"ID":"47c57138-56c1-4545-8369-d5a726afbdf5","Type":"ContainerDied","Data":"af36c08bf4a5b450479c3f2eedd5a87d5487275fe602a661c0251d85dcdc5700"}
Sep 16 19:01:23.304686 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:23.304430    2580 scope.go:117] "RemoveContainer" containerID="a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d"
Sep 16 19:01:23.314004 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:23.313985    2580 scope.go:117] "RemoveContainer" containerID="a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d"
Sep 16 19:01:23.314281 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:01:23.314263    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d\": container with ID starting with a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d not found: ID does not exist" containerID="a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d"
Sep 16 19:01:23.314330 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:23.314288    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d"} err="failed to get container status \"a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d\": rpc error: code = NotFound desc = could not find container \"a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d\": container with ID starting with a4d2a98f627d89958bddddac577335c516b0d73faa05a4f30f82284a5349d49d not found: ID does not exist"
Sep 16 19:01:23.326698 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:23.326677    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"]
Sep 16 19:01:23.330555 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:23.330536    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f94l5l"]
Sep 16 19:01:24.275858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:01:24.275822    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47c57138-56c1-4545-8369-d5a726afbdf5" path="/var/lib/kubelet/pods/47c57138-56c1-4545-8369-d5a726afbdf5/volumes"
Sep 16 19:02:20.417756 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.417722    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"]
Sep 16 19:02:20.418271 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.418118    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c57138-56c1-4545-8369-d5a726afbdf5" containerName="main"
Sep 16 19:02:20.418271 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.418131    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c57138-56c1-4545-8369-d5a726afbdf5" containerName="main"
Sep 16 19:02:20.418271 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.418196    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="47c57138-56c1-4545-8369-d5a726afbdf5" containerName="main"
Sep 16 19:02:20.420383 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.420364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.423329 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.423293    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 19:02:20.424096 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.424073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 19:02:20.424208 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.424078    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 19:02:20.424593 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.424576    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-8tkjm\""
Sep 16 19:02:20.437606 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.437564    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"]
Sep 16 19:02:20.563511 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.563458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.563511 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.563506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.563755 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.563601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def52f0a-dbac-42af-8374-05d8e9d523f8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.563755 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.563633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.563755 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.563673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.563755 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.563708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvbg8\" (UniqueName: \"kubernetes.io/projected/def52f0a-dbac-42af-8374-05d8e9d523f8-kube-api-access-zvbg8\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.664804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.664765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.664804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.664809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbg8\" (UniqueName: \"kubernetes.io/projected/def52f0a-dbac-42af-8374-05d8e9d523f8-kube-api-access-zvbg8\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.665125 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.664951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.665125 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.664991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.665125 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.665057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def52f0a-dbac-42af-8374-05d8e9d523f8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.665125 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.665074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.665339 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.665275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.665406 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.665350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.665457 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.665413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.667596 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.667571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.667712 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.667694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def52f0a-dbac-42af-8374-05d8e9d523f8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.679613 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.679572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbg8\" (UniqueName: \"kubernetes.io/projected/def52f0a-dbac-42af-8374-05d8e9d523f8-kube-api-access-zvbg8\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:20.731546 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:20.731503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:21.008701 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:21.008673    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"]
Sep 16 19:02:21.011247 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:02:21.011217    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef52f0a_dbac_42af_8374_05d8e9d523f8.slice/crio-2742dd3109b7037182e9c9ead4377fac8d752b2043c3d1966d4f85a1bc36cdd4 WatchSource:0}: Error finding container 2742dd3109b7037182e9c9ead4377fac8d752b2043c3d1966d4f85a1bc36cdd4: Status 404 returned error can't find the container with id 2742dd3109b7037182e9c9ead4377fac8d752b2043c3d1966d4f85a1bc36cdd4
Sep 16 19:02:21.584181 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:21.584096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg" event={"ID":"def52f0a-dbac-42af-8374-05d8e9d523f8","Type":"ContainerStarted","Data":"14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b"}
Sep 16 19:02:21.584963 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:21.584238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg" event={"ID":"def52f0a-dbac-42af-8374-05d8e9d523f8","Type":"ContainerStarted","Data":"2742dd3109b7037182e9c9ead4377fac8d752b2043c3d1966d4f85a1bc36cdd4"}
Sep 16 19:02:21.610826 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:21.610680    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg" podStartSLOduration=1.6106374159999999 podStartE2EDuration="1.610637416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:21.606535088 +0000 UTC m=+1121.868212092" watchObservedRunningTime="2026-09-16 19:02:21.610637416 +0000 UTC m=+1121.872314431"
Sep 16 19:02:21.732013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:21.731975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:30.732557 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:30.732507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:30.745338 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:30.745302    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:31.642190 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:31.642153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:54.596192 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:54.596155    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"]
Sep 16 19:02:54.606585 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:02:54.606539    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:02:54.606585 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:02:54.606585 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:02:54.606585 ip-10-0-139-95 kubenswrapper[2580]:  > containerID="14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b" cmd=["/bin/sleep","15"]
Sep 16 19:02:54.606956 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:54.606602    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg" message="" err=<
Sep 16 19:02:54.606956 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:02:54.606956 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:02:54.606956 ip-10-0-139-95 kubenswrapper[2580]:  >
Sep 16 19:02:54.606956 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:02:54.606620    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:02:54.606956 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:02:54.606956 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:02:54.606956 ip-10-0-139-95 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg" podUID="def52f0a-dbac-42af-8374-05d8e9d523f8" containerName="main" containerID="cri-o://14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b"
Sep 16 19:02:54.606956 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:54.606656    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg" podUID="def52f0a-dbac-42af-8374-05d8e9d523f8" containerName="main" containerID="cri-o://14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b" gracePeriod=60
Sep 16 19:02:54.631279 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:54.631228    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg" podUID="def52f0a-dbac-42af-8374-05d8e9d523f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 19:02:54.954090 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:54.954060    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:55.074290 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074252    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-model-cache\") pod \"def52f0a-dbac-42af-8374-05d8e9d523f8\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") "
Sep 16 19:02:55.074290 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074296    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def52f0a-dbac-42af-8374-05d8e9d523f8-tls-certs\") pod \"def52f0a-dbac-42af-8374-05d8e9d523f8\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") "
Sep 16 19:02:55.074519 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-tmp-dir\") pod \"def52f0a-dbac-42af-8374-05d8e9d523f8\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") "
Sep 16 19:02:55.074569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074524    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-dshm\") pod \"def52f0a-dbac-42af-8374-05d8e9d523f8\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") "
Sep 16 19:02:55.074569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074564    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "def52f0a-dbac-42af-8374-05d8e9d523f8" (UID: "def52f0a-dbac-42af-8374-05d8e9d523f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:55.074682 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "def52f0a-dbac-42af-8374-05d8e9d523f8" (UID: "def52f0a-dbac-42af-8374-05d8e9d523f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:55.074682 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvbg8\" (UniqueName: \"kubernetes.io/projected/def52f0a-dbac-42af-8374-05d8e9d523f8-kube-api-access-zvbg8\") pod \"def52f0a-dbac-42af-8374-05d8e9d523f8\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") "
Sep 16 19:02:55.074682 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074637    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-home\") pod \"def52f0a-dbac-42af-8374-05d8e9d523f8\" (UID: \"def52f0a-dbac-42af-8374-05d8e9d523f8\") "
Sep 16 19:02:55.074880 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074862    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:02:55.074981 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074884    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:02:55.074981 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.074924    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-home" (OuterVolumeSpecName: "home") pod "def52f0a-dbac-42af-8374-05d8e9d523f8" (UID: "def52f0a-dbac-42af-8374-05d8e9d523f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:55.076855 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.076825    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-dshm" (OuterVolumeSpecName: "dshm") pod "def52f0a-dbac-42af-8374-05d8e9d523f8" (UID: "def52f0a-dbac-42af-8374-05d8e9d523f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:55.076982 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.076881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def52f0a-dbac-42af-8374-05d8e9d523f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "def52f0a-dbac-42af-8374-05d8e9d523f8" (UID: "def52f0a-dbac-42af-8374-05d8e9d523f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:55.076982 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.076881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def52f0a-dbac-42af-8374-05d8e9d523f8-kube-api-access-zvbg8" (OuterVolumeSpecName: "kube-api-access-zvbg8") pod "def52f0a-dbac-42af-8374-05d8e9d523f8" (UID: "def52f0a-dbac-42af-8374-05d8e9d523f8"). InnerVolumeSpecName "kube-api-access-zvbg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:55.176337 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.176300    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:02:55.176337 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.176336    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvbg8\" (UniqueName: \"kubernetes.io/projected/def52f0a-dbac-42af-8374-05d8e9d523f8-kube-api-access-zvbg8\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:02:55.176563 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.176352    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def52f0a-dbac-42af-8374-05d8e9d523f8-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:02:55.176563 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.176363    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def52f0a-dbac-42af-8374-05d8e9d523f8-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:02:55.748640 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.748606    2580 generic.go:356] "Generic (PLEG): container finished" podID="def52f0a-dbac-42af-8374-05d8e9d523f8" containerID="14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b" exitCode=0
Sep 16 19:02:55.749164 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.748663    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"
Sep 16 19:02:55.749164 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.748683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg" event={"ID":"def52f0a-dbac-42af-8374-05d8e9d523f8","Type":"ContainerDied","Data":"14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b"}
Sep 16 19:02:55.749164 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.748726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg" event={"ID":"def52f0a-dbac-42af-8374-05d8e9d523f8","Type":"ContainerDied","Data":"2742dd3109b7037182e9c9ead4377fac8d752b2043c3d1966d4f85a1bc36cdd4"}
Sep 16 19:02:55.749164 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.748750    2580 scope.go:117] "RemoveContainer" containerID="14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b"
Sep 16 19:02:55.758975 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.758950    2580 scope.go:117] "RemoveContainer" containerID="14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b"
Sep 16 19:02:55.759354 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:02:55.759335    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b\": container with ID starting with 14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b not found: ID does not exist" containerID="14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b"
Sep 16 19:02:55.759413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.759366    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b"} err="failed to get container status \"14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b\": rpc error: code = NotFound desc = could not find container \"14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b\": container with ID starting with 14cc268ee1d122f26c35188375904867f3f2d2bd45bbb6196e3928bc949eb18b not found: ID does not exist"
Sep 16 19:02:55.773186 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.773151    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"]
Sep 16 19:02:55.777137 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:55.777109    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cj4zg"]
Sep 16 19:02:56.275828 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:02:56.275789    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def52f0a-dbac-42af-8374-05d8e9d523f8" path="/var/lib/kubelet/pods/def52f0a-dbac-42af-8374-05d8e9d523f8/volumes"
Sep 16 19:03:08.923256 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.923181    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"]
Sep 16 19:03:08.923653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.923552    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def52f0a-dbac-42af-8374-05d8e9d523f8" containerName="main"
Sep 16 19:03:08.923653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.923566    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="def52f0a-dbac-42af-8374-05d8e9d523f8" containerName="main"
Sep 16 19:03:08.923653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.923636    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="def52f0a-dbac-42af-8374-05d8e9d523f8" containerName="main"
Sep 16 19:03:08.926599 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.926574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:08.929293 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.929261    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:08.929423 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.929361    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:08.929423 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.929377    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 19:03:08.929423 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.929408    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 19:03:08.929600 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.929518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-zkcrz\""
Sep 16 19:03:08.929653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.929605    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 19:03:08.942849 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:08.942821    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"]
Sep 16 19:03:09.106842 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.106804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-model-cache\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.106842 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.106843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.107105 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.106863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e320467-ed51-4668-96ae-90b6b0d6feb6-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.107105 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.107004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdv5\" (UniqueName: \"kubernetes.io/projected/4e320467-ed51-4668-96ae-90b6b0d6feb6-kube-api-access-hfdv5\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.107105 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.107057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-tmp-dir\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.107210 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.107104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-dshm\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.107210 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.107129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-home\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.107210 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.107164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e320467-ed51-4668-96ae-90b6b0d6feb6-tls-certs\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.170764 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.170724    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"]
Sep 16 19:03:09.174179 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.174116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:09.176743 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.176716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-w99bz\""
Sep 16 19:03:09.187208 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.187170    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"]
Sep 16 19:03:09.207794 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.207750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-tmp-dir\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208008 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.207806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-dshm\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208008 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.207828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-home\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208008 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.207862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e320467-ed51-4668-96ae-90b6b0d6feb6-tls-certs\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208008 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.207940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-model-cache\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208008 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.207964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208008 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.207991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e320467-ed51-4668-96ae-90b6b0d6feb6-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208353 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.208037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdv5\" (UniqueName: \"kubernetes.io/projected/4e320467-ed51-4668-96ae-90b6b0d6feb6-kube-api-access-hfdv5\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208353 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.208233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-tmp-dir\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208353 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.208311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-home\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208537 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.208516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208603 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.208585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-model-cache\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.208780 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.208759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e320467-ed51-4668-96ae-90b6b0d6feb6-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.210350 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.210332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-dshm\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.210601 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.210586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e320467-ed51-4668-96ae-90b6b0d6feb6-tls-certs\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.218437 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.218412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdv5\" (UniqueName: \"kubernetes.io/projected/4e320467-ed51-4668-96ae-90b6b0d6feb6-kube-api-access-hfdv5\") pod \"auth-invalid-token-test-kserve-5744f59bbf-t2dj9\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.238221 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.238171    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:09.309690 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.309635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdh5\" (UniqueName: \"kubernetes.io/projected/c06a2f6b-9cba-4222-9589-6308b182ee39-kube-api-access-zkdh5\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx\" (UID: \"c06a2f6b-9cba-4222-9589-6308b182ee39\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:09.309934 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.309824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c06a2f6b-9cba-4222-9589-6308b182ee39-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx\" (UID: \"c06a2f6b-9cba-4222-9589-6308b182ee39\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:09.411452 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.411384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c06a2f6b-9cba-4222-9589-6308b182ee39-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx\" (UID: \"c06a2f6b-9cba-4222-9589-6308b182ee39\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:09.411685 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.411503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdh5\" (UniqueName: \"kubernetes.io/projected/c06a2f6b-9cba-4222-9589-6308b182ee39-kube-api-access-zkdh5\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx\" (UID: \"c06a2f6b-9cba-4222-9589-6308b182ee39\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:09.414179 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.414140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c06a2f6b-9cba-4222-9589-6308b182ee39-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx\" (UID: \"c06a2f6b-9cba-4222-9589-6308b182ee39\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:09.419795 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.419763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdh5\" (UniqueName: \"kubernetes.io/projected/c06a2f6b-9cba-4222-9589-6308b182ee39-kube-api-access-zkdh5\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx\" (UID: \"c06a2f6b-9cba-4222-9589-6308b182ee39\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:09.485752 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.485660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:09.532836 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.532672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"]
Sep 16 19:03:09.536109 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:03:09.536076    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e320467_ed51_4668_96ae_90b6b0d6feb6.slice/crio-d68070f704cc02f4ce335c9c83992002ef5840fe5d5113d36c1afacb7749d3ef WatchSource:0}: Error finding container d68070f704cc02f4ce335c9c83992002ef5840fe5d5113d36c1afacb7749d3ef: Status 404 returned error can't find the container with id d68070f704cc02f4ce335c9c83992002ef5840fe5d5113d36c1afacb7749d3ef
Sep 16 19:03:09.768003 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.767969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"]
Sep 16 19:03:09.769859 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:03:09.769818    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06a2f6b_9cba_4222_9589_6308b182ee39.slice/crio-5207a1f1dd179294d4f0bfbab9f564527580987e2db4820695a2f4fe21dbf1b9 WatchSource:0}: Error finding container 5207a1f1dd179294d4f0bfbab9f564527580987e2db4820695a2f4fe21dbf1b9: Status 404 returned error can't find the container with id 5207a1f1dd179294d4f0bfbab9f564527580987e2db4820695a2f4fe21dbf1b9
Sep 16 19:03:09.821719 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.821680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" event={"ID":"c06a2f6b-9cba-4222-9589-6308b182ee39","Type":"ContainerStarted","Data":"5207a1f1dd179294d4f0bfbab9f564527580987e2db4820695a2f4fe21dbf1b9"}
Sep 16 19:03:09.823612 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.823572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" event={"ID":"4e320467-ed51-4668-96ae-90b6b0d6feb6","Type":"ContainerStarted","Data":"ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9"}
Sep 16 19:03:09.823754 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:09.823639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" event={"ID":"4e320467-ed51-4668-96ae-90b6b0d6feb6","Type":"ContainerStarted","Data":"d68070f704cc02f4ce335c9c83992002ef5840fe5d5113d36c1afacb7749d3ef"}
Sep 16 19:03:10.830208 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:10.830167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" event={"ID":"c06a2f6b-9cba-4222-9589-6308b182ee39","Type":"ContainerStarted","Data":"acc8d78b53b9094c26c819e8928ca4ecd47573b16898e42805887417fb5b7359"}
Sep 16 19:03:10.830612 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:10.830415    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:10.850174 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:10.850096    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" podStartSLOduration=1.850069477 podStartE2EDuration="1.850069477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:10.84670674 +0000 UTC m=+1171.108383766" watchObservedRunningTime="2026-09-16 19:03:10.850069477 +0000 UTC m=+1171.111746491"
Sep 16 19:03:12.842407 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:12.842373    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerID="ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9" exitCode=0
Sep 16 19:03:12.842407 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:12.842411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" event={"ID":"4e320467-ed51-4668-96ae-90b6b0d6feb6","Type":"ContainerDied","Data":"ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9"}
Sep 16 19:03:13.850089 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:13.850051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" event={"ID":"4e320467-ed51-4668-96ae-90b6b0d6feb6","Type":"ContainerStarted","Data":"5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8"}
Sep 16 19:03:13.874682 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:13.874609    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podStartSLOduration=3.473702557 podStartE2EDuration="5.874584377s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.40088182s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:13.873499788 +0000 UTC m=+1174.135176800" watchObservedRunningTime="2026-09-16 19:03:13.874584377 +0000 UTC m=+1174.136261391"
Sep 16 19:03:14.238983 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:14.238943    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:19.238872 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:19.238830    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:03:19.240562 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:19.240532    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:03:29.239612 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:29.239558    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:03:39.239016 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:39.238960    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:03:41.837080 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:41.837045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:03:49.239141 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:49.239086    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:03:52.593333 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.593282    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"]
Sep 16 19:03:52.598398 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.598372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:03:52.600763 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.600735    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:03:52.601002 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.600830    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-rcjxx\""
Sep 16 19:03:52.601561 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.601541    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:03:52.601670 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.601562    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-hr6fv\""
Sep 16 19:03:52.601735 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.601683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:03:52.609701 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.609673    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"]
Sep 16 19:03:52.622693 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.622649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzfnl\" (UniqueName: \"kubernetes.io/projected/dbd07a6d-bae0-41bb-a4b2-621944a8351c-kube-api-access-hzfnl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b\" (UID: \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:03:52.622949 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.622835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd07a6d-bae0-41bb-a4b2-621944a8351c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b\" (UID: \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:03:52.723426 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.723385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd07a6d-bae0-41bb-a4b2-621944a8351c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b\" (UID: \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:03:52.723647 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.723495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzfnl\" (UniqueName: \"kubernetes.io/projected/dbd07a6d-bae0-41bb-a4b2-621944a8351c-kube-api-access-hzfnl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b\" (UID: \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:03:52.726382 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.726338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd07a6d-bae0-41bb-a4b2-621944a8351c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b\" (UID: \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:03:52.734355 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.734320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzfnl\" (UniqueName: \"kubernetes.io/projected/dbd07a6d-bae0-41bb-a4b2-621944a8351c-kube-api-access-hzfnl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b\" (UID: \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:03:52.912456 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:52.912275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:03:53.206708 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:53.206677    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"]
Sep 16 19:03:53.209245 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:03:53.209210    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd07a6d_bae0_41bb_a4b2_621944a8351c.slice/crio-67e6d3c13b4a2120cf127e81b4d3f17191b152f17017690f7f6b0c320a4548d7 WatchSource:0}: Error finding container 67e6d3c13b4a2120cf127e81b4d3f17191b152f17017690f7f6b0c320a4548d7: Status 404 returned error can't find the container with id 67e6d3c13b4a2120cf127e81b4d3f17191b152f17017690f7f6b0c320a4548d7
Sep 16 19:03:54.068160 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:54.068118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" event={"ID":"dbd07a6d-bae0-41bb-a4b2-621944a8351c","Type":"ContainerStarted","Data":"effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f"}
Sep 16 19:03:54.068690 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:54.068179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" event={"ID":"dbd07a6d-bae0-41bb-a4b2-621944a8351c","Type":"ContainerStarted","Data":"67e6d3c13b4a2120cf127e81b4d3f17191b152f17017690f7f6b0c320a4548d7"}
Sep 16 19:03:54.068690 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:54.068206    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:03:54.085217 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:54.085145    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" podStartSLOduration=2.085121573 podStartE2EDuration="2.085121573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:54.083814975 +0000 UTC m=+1214.345491989" watchObservedRunningTime="2026-09-16 19:03:54.085121573 +0000 UTC m=+1214.346798588"
Sep 16 19:03:59.239471 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:03:59.239428    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:09.239713 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:09.239654    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:19.239130 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:19.239081    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:25.075874 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:25.075828    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:25.995467 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:25.995423    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"]
Sep 16 19:04:25.999943 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:25.999843    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:29.239325 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:29.239274    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:35.998764 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:35.998723    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:39.239628 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:39.239586    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:41.024714 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:41.024670    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main" containerID="cri-o://effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f" gracePeriod=45
Sep 16 19:04:45.998357 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:45.998319    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:49.238882 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:49.238838    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:55.998637 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:55.998588    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:59.239543 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:04:59.239497    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:05:05.998166 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:05.998119    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:09.238802 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:09.238742    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:05:11.313188 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.313156    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:05:11.463150 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.463089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzfnl\" (UniqueName: \"kubernetes.io/projected/dbd07a6d-bae0-41bb-a4b2-621944a8351c-kube-api-access-hzfnl\") pod \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\" (UID: \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\") "
Sep 16 19:05:11.463392 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.463164    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd07a6d-bae0-41bb-a4b2-621944a8351c-tls-certs\") pod \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\" (UID: \"dbd07a6d-bae0-41bb-a4b2-621944a8351c\") "
Sep 16 19:05:11.466823 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.466764    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd07a6d-bae0-41bb-a4b2-621944a8351c-kube-api-access-hzfnl" (OuterVolumeSpecName: "kube-api-access-hzfnl") pod "dbd07a6d-bae0-41bb-a4b2-621944a8351c" (UID: "dbd07a6d-bae0-41bb-a4b2-621944a8351c"). InnerVolumeSpecName "kube-api-access-hzfnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:11.467786 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.467749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd07a6d-bae0-41bb-a4b2-621944a8351c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbd07a6d-bae0-41bb-a4b2-621944a8351c" (UID: "dbd07a6d-bae0-41bb-a4b2-621944a8351c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:11.473470 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.473440    2580 generic.go:356] "Generic (PLEG): container finished" podID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerID="effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f" exitCode=0
Sep 16 19:05:11.473645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.473501    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"
Sep 16 19:05:11.473645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.473525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" event={"ID":"dbd07a6d-bae0-41bb-a4b2-621944a8351c","Type":"ContainerDied","Data":"effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f"}
Sep 16 19:05:11.473645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.473591    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b" event={"ID":"dbd07a6d-bae0-41bb-a4b2-621944a8351c","Type":"ContainerDied","Data":"67e6d3c13b4a2120cf127e81b4d3f17191b152f17017690f7f6b0c320a4548d7"}
Sep 16 19:05:11.473645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.473621    2580 scope.go:117] "RemoveContainer" containerID="effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f"
Sep 16 19:05:11.493693 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.493639    2580 scope.go:117] "RemoveContainer" containerID="effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f"
Sep 16 19:05:11.494269 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:05:11.494237    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f\": container with ID starting with effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f not found: ID does not exist" containerID="effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f"
Sep 16 19:05:11.494389 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.494282    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f"} err="failed to get container status \"effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f\": rpc error: code = NotFound desc = could not find container \"effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f\": container with ID starting with effda2d4899f5d3ad6abb1eeaace32c469af03d38b125b4d1b5460a10b4eee8f not found: ID does not exist"
Sep 16 19:05:11.506090 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.506035    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"]
Sep 16 19:05:11.510884 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.510852    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5frg6b"]
Sep 16 19:05:11.564767 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.564652    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd07a6d-bae0-41bb-a4b2-621944a8351c-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:11.564767 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:11.564699    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzfnl\" (UniqueName: \"kubernetes.io/projected/dbd07a6d-bae0-41bb-a4b2-621944a8351c-kube-api-access-hzfnl\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:12.281565 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:12.281517    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" path="/var/lib/kubelet/pods/dbd07a6d-bae0-41bb-a4b2-621944a8351c/volumes"
Sep 16 19:05:19.250527 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:19.250478    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:05:19.261385 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:19.261338    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:05:38.582905 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:38.582836    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"]
Sep 16 19:05:38.593954 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:38.593921    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"]
Sep 16 19:05:41.837782 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:41.837736    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:51.838302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:51.838256    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:53.605264 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:53.605201    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerName="main" containerID="cri-o://acc8d78b53b9094c26c819e8928ca4ecd47573b16898e42805887417fb5b7359" gracePeriod=45
Sep 16 19:05:53.619326 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:53.619264    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" containerID="cri-o://5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8" gracePeriod=45
Sep 16 19:05:54.251627 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.251582    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:05:54.375568 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.375528    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:05:54.464788 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.464747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e320467-ed51-4668-96ae-90b6b0d6feb6-tls-certs\") pod \"4e320467-ed51-4668-96ae-90b6b0d6feb6\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") "
Sep 16 19:05:54.464981 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.464827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-kserve-provision-location\") pod \"4e320467-ed51-4668-96ae-90b6b0d6feb6\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") "
Sep 16 19:05:54.464981 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.464955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-tmp-dir\") pod \"4e320467-ed51-4668-96ae-90b6b0d6feb6\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") "
Sep 16 19:05:54.465099 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.464995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-dshm\") pod \"4e320467-ed51-4668-96ae-90b6b0d6feb6\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") "
Sep 16 19:05:54.465099 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.465026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-model-cache\") pod \"4e320467-ed51-4668-96ae-90b6b0d6feb6\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") "
Sep 16 19:05:54.465099 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.465052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfdv5\" (UniqueName: \"kubernetes.io/projected/4e320467-ed51-4668-96ae-90b6b0d6feb6-kube-api-access-hfdv5\") pod \"4e320467-ed51-4668-96ae-90b6b0d6feb6\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") "
Sep 16 19:05:54.465099 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.465074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-home\") pod \"4e320467-ed51-4668-96ae-90b6b0d6feb6\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") "
Sep 16 19:05:54.465305 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.465134    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e320467-ed51-4668-96ae-90b6b0d6feb6-cabundle-cert\") pod \"4e320467-ed51-4668-96ae-90b6b0d6feb6\" (UID: \"4e320467-ed51-4668-96ae-90b6b0d6feb6\") "
Sep 16 19:05:54.465364 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.465324    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e320467-ed51-4668-96ae-90b6b0d6feb6" (UID: "4e320467-ed51-4668-96ae-90b6b0d6feb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:54.465483 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.465463    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:54.465859 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.465830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-home" (OuterVolumeSpecName: "home") pod "4e320467-ed51-4668-96ae-90b6b0d6feb6" (UID: "4e320467-ed51-4668-96ae-90b6b0d6feb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:54.466006 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.465833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e320467-ed51-4668-96ae-90b6b0d6feb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e320467-ed51-4668-96ae-90b6b0d6feb6" (UID: "4e320467-ed51-4668-96ae-90b6b0d6feb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:54.467509 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.467481    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e320467-ed51-4668-96ae-90b6b0d6feb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e320467-ed51-4668-96ae-90b6b0d6feb6" (UID: "4e320467-ed51-4668-96ae-90b6b0d6feb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:54.467619 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.467511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e320467-ed51-4668-96ae-90b6b0d6feb6-kube-api-access-hfdv5" (OuterVolumeSpecName: "kube-api-access-hfdv5") pod "4e320467-ed51-4668-96ae-90b6b0d6feb6" (UID: "4e320467-ed51-4668-96ae-90b6b0d6feb6"). InnerVolumeSpecName "kube-api-access-hfdv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:54.467673 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.467632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-dshm" (OuterVolumeSpecName: "dshm") pod "4e320467-ed51-4668-96ae-90b6b0d6feb6" (UID: "4e320467-ed51-4668-96ae-90b6b0d6feb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:54.477302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.477267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e320467-ed51-4668-96ae-90b6b0d6feb6" (UID: "4e320467-ed51-4668-96ae-90b6b0d6feb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:54.487455 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.487419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e320467-ed51-4668-96ae-90b6b0d6feb6" (UID: "4e320467-ed51-4668-96ae-90b6b0d6feb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:54.566311 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.566270    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:54.566311 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.566306    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfdv5\" (UniqueName: \"kubernetes.io/projected/4e320467-ed51-4668-96ae-90b6b0d6feb6-kube-api-access-hfdv5\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:54.566311 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.566317    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:54.566525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.566325    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e320467-ed51-4668-96ae-90b6b0d6feb6-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:54.566525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.566335    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e320467-ed51-4668-96ae-90b6b0d6feb6-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:54.566525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.566343    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:54.566525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.566352    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e320467-ed51-4668-96ae-90b6b0d6feb6-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:05:54.687050 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.687009    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerID="5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8" exitCode=0
Sep 16 19:05:54.687413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.687060    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"
Sep 16 19:05:54.687413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.687072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" event={"ID":"4e320467-ed51-4668-96ae-90b6b0d6feb6","Type":"ContainerDied","Data":"5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8"}
Sep 16 19:05:54.687413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.687124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9" event={"ID":"4e320467-ed51-4668-96ae-90b6b0d6feb6","Type":"ContainerDied","Data":"d68070f704cc02f4ce335c9c83992002ef5840fe5d5113d36c1afacb7749d3ef"}
Sep 16 19:05:54.687413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.687151    2580 scope.go:117] "RemoveContainer" containerID="5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8"
Sep 16 19:05:54.696774 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.696739    2580 scope.go:117] "RemoveContainer" containerID="ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9"
Sep 16 19:05:54.705850 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.705825    2580 scope.go:117] "RemoveContainer" containerID="5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8"
Sep 16 19:05:54.706235 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:05:54.706215    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8\": container with ID starting with 5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8 not found: ID does not exist" containerID="5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8"
Sep 16 19:05:54.706299 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.706246    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8"} err="failed to get container status \"5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8\": rpc error: code = NotFound desc = could not find container \"5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8\": container with ID starting with 5a35ef43a8a547029c2dd8d7493ffc3cea1f7780a38e6f4d3125deac2f26fcc8 not found: ID does not exist"
Sep 16 19:05:54.706299 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.706267    2580 scope.go:117] "RemoveContainer" containerID="ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9"
Sep 16 19:05:54.706565 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:05:54.706546    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9\": container with ID starting with ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9 not found: ID does not exist" containerID="ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9"
Sep 16 19:05:54.706610 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.706572    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9"} err="failed to get container status \"ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9\": rpc error: code = NotFound desc = could not find container \"ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9\": container with ID starting with ccd7036d0cec42bfed25a2ec3f8b27ad7ee1ed63466c06169aed9fa9cbcd63c9 not found: ID does not exist"
Sep 16 19:05:54.711211 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.711180    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"]
Sep 16 19:05:54.715420 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:54.715358    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5744f59bbf-t2dj9"]
Sep 16 19:05:56.275963 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:05:56.275925    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" path="/var/lib/kubelet/pods/4e320467-ed51-4668-96ae-90b6b0d6feb6/volumes"
Sep 16 19:06:01.837929 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:01.837817    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:01.838325 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:01.837949    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:06:11.837298 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:11.837253    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:21.836793 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:21.836753    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:23.825199 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:23.825170    2580 generic.go:356] "Generic (PLEG): container finished" podID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerID="acc8d78b53b9094c26c819e8928ca4ecd47573b16898e42805887417fb5b7359" exitCode=0
Sep 16 19:06:23.825610 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:23.825225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" event={"ID":"c06a2f6b-9cba-4222-9589-6308b182ee39","Type":"ContainerDied","Data":"acc8d78b53b9094c26c819e8928ca4ecd47573b16898e42805887417fb5b7359"}
Sep 16 19:06:23.847767 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:23.847744    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:06:23.918845 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:23.918787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkdh5\" (UniqueName: \"kubernetes.io/projected/c06a2f6b-9cba-4222-9589-6308b182ee39-kube-api-access-zkdh5\") pod \"c06a2f6b-9cba-4222-9589-6308b182ee39\" (UID: \"c06a2f6b-9cba-4222-9589-6308b182ee39\") "
Sep 16 19:06:23.918845 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:23.918843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c06a2f6b-9cba-4222-9589-6308b182ee39-tls-certs\") pod \"c06a2f6b-9cba-4222-9589-6308b182ee39\" (UID: \"c06a2f6b-9cba-4222-9589-6308b182ee39\") "
Sep 16 19:06:23.920916 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:23.920871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06a2f6b-9cba-4222-9589-6308b182ee39-kube-api-access-zkdh5" (OuterVolumeSpecName: "kube-api-access-zkdh5") pod "c06a2f6b-9cba-4222-9589-6308b182ee39" (UID: "c06a2f6b-9cba-4222-9589-6308b182ee39"). InnerVolumeSpecName "kube-api-access-zkdh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:23.921011 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:23.920994    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06a2f6b-9cba-4222-9589-6308b182ee39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c06a2f6b-9cba-4222-9589-6308b182ee39" (UID: "c06a2f6b-9cba-4222-9589-6308b182ee39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:24.019985 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:24.019960    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c06a2f6b-9cba-4222-9589-6308b182ee39-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:06:24.019985 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:24.019982    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkdh5\" (UniqueName: \"kubernetes.io/projected/c06a2f6b-9cba-4222-9589-6308b182ee39-kube-api-access-zkdh5\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:06:24.831037 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:24.830997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx" event={"ID":"c06a2f6b-9cba-4222-9589-6308b182ee39","Type":"ContainerDied","Data":"5207a1f1dd179294d4f0bfbab9f564527580987e2db4820695a2f4fe21dbf1b9"}
Sep 16 19:06:24.831473 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:24.831049    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"
Sep 16 19:06:24.831473 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:24.831103    2580 scope.go:117] "RemoveContainer" containerID="acc8d78b53b9094c26c819e8928ca4ecd47573b16898e42805887417fb5b7359"
Sep 16 19:06:24.849656 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:24.849618    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"]
Sep 16 19:06:24.853330 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:24.853306    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9czwhxx"]
Sep 16 19:06:26.275712 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:26.275676    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" path="/var/lib/kubelet/pods/c06a2f6b-9cba-4222-9589-6308b182ee39/volumes"
Sep 16 19:06:57.838795 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.838764    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"]
Sep 16 19:06:57.839302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839222    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main"
Sep 16 19:06:57.839302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839238    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main"
Sep 16 19:06:57.839302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839254    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="storage-initializer"
Sep 16 19:06:57.839302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839260    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="storage-initializer"
Sep 16 19:06:57.839302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839269    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main"
Sep 16 19:06:57.839302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839274    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main"
Sep 16 19:06:57.839302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839290    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerName="main"
Sep 16 19:06:57.839302 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839297    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerName="main"
Sep 16 19:06:57.839554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839363    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c06a2f6b-9cba-4222-9589-6308b182ee39" containerName="main"
Sep 16 19:06:57.839554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839371    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e320467-ed51-4668-96ae-90b6b0d6feb6" containerName="main"
Sep 16 19:06:57.839554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.839380    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd07a6d-bae0-41bb-a4b2-621944a8351c" containerName="main"
Sep 16 19:06:57.843843 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.843826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:57.846727 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.846678    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-sv6vn\""
Sep 16 19:06:57.846727 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.846722    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:06:57.846961 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.846740    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:06:57.846961 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.846754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:06:57.846961 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.846756    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 19:06:57.847104 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.846969    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:06:57.861741 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.861720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"]
Sep 16 19:06:57.991114 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.991075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1249f95-33b7-488c-b636-e3020a24fb16-cabundle-cert\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:57.991114 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.991121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-model-cache\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:57.991311 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.991143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-tmp-dir\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:57.991311 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.991196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-dshm\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:57.991311 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.991224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1249f95-33b7-488c-b636-e3020a24fb16-tls-certs\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:57.991311 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.991250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-home\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:57.991311 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.991272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-kserve-provision-location\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:57.991311 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:57.991295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjpsr\" (UniqueName: \"kubernetes.io/projected/b1249f95-33b7-488c-b636-e3020a24fb16-kube-api-access-hjpsr\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092415 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-home\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092415 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-kserve-provision-location\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092415 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjpsr\" (UniqueName: \"kubernetes.io/projected/b1249f95-33b7-488c-b636-e3020a24fb16-kube-api-access-hjpsr\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1249f95-33b7-488c-b636-e3020a24fb16-cabundle-cert\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-model-cache\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-tmp-dir\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-dshm\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1249f95-33b7-488c-b636-e3020a24fb16-tls-certs\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092934 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-home\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.092934 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-kserve-provision-location\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.093050 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.092983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-tmp-dir\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.093050 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.093024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-model-cache\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.093621 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.093603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1249f95-33b7-488c-b636-e3020a24fb16-cabundle-cert\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.095286 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.095263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-dshm\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.095643 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.095611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1249f95-33b7-488c-b636-e3020a24fb16-tls-certs\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.114677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.114655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjpsr\" (UniqueName: \"kubernetes.io/projected/b1249f95-33b7-488c-b636-e3020a24fb16-kube-api-access-hjpsr\") pod \"auth-disabled-test-kserve-64d4875f8-xrs6k\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.155431 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.155402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:06:58.443075 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.443051    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"]
Sep 16 19:06:58.444655 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:06:58.444630    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1249f95_33b7_488c_b636_e3020a24fb16.slice/crio-9f4e89cacb6fee4b8ce0045ad91fae1ef815f474370f8f496514bc44185be2d9 WatchSource:0}: Error finding container 9f4e89cacb6fee4b8ce0045ad91fae1ef815f474370f8f496514bc44185be2d9: Status 404 returned error can't find the container with id 9f4e89cacb6fee4b8ce0045ad91fae1ef815f474370f8f496514bc44185be2d9
Sep 16 19:06:58.991932 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.991875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" event={"ID":"b1249f95-33b7-488c-b636-e3020a24fb16","Type":"ContainerStarted","Data":"e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f"}
Sep 16 19:06:58.992314 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:06:58.991946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" event={"ID":"b1249f95-33b7-488c-b636-e3020a24fb16","Type":"ContainerStarted","Data":"9f4e89cacb6fee4b8ce0045ad91fae1ef815f474370f8f496514bc44185be2d9"}
Sep 16 19:07:01.003036 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:01.002998    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1249f95-33b7-488c-b636-e3020a24fb16" containerID="e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f" exitCode=0
Sep 16 19:07:01.003384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:01.003075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" event={"ID":"b1249f95-33b7-488c-b636-e3020a24fb16","Type":"ContainerDied","Data":"e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f"}
Sep 16 19:07:02.009458 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:02.009423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" event={"ID":"b1249f95-33b7-488c-b636-e3020a24fb16","Type":"ContainerStarted","Data":"ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23"}
Sep 16 19:07:02.034960 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:02.034879    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podStartSLOduration=2.885602301 podStartE2EDuration="5.034858754s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.149256453s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:02.034389221 +0000 UTC m=+1402.296066234" watchObservedRunningTime="2026-09-16 19:07:02.034858754 +0000 UTC m=+1402.296535770"
Sep 16 19:07:02.155803 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:02.155771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:07:08.155620 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:08.155582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:07:08.157307 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:08.157279    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:07:18.156111 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:18.156065    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:07:25.480792 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.480761    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"]
Sep 16 19:07:25.483350 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.483329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:25.486218 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.486188    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:07:25.486562 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.486409    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:07:25.486562 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.486440    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-pcphf\""
Sep 16 19:07:25.486562 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.486529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-9thct\""
Sep 16 19:07:25.487011 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.486993    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:07:25.494491 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.494467    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"]
Sep 16 19:07:25.534151 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.534110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z\" (UID: \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:25.534312 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.534157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkxf\" (UniqueName: \"kubernetes.io/projected/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-kube-api-access-4gkxf\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z\" (UID: \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:25.634772 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.634727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z\" (UID: \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:25.634772 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.634761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkxf\" (UniqueName: \"kubernetes.io/projected/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-kube-api-access-4gkxf\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z\" (UID: \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:25.637480 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.637448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z\" (UID: \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:25.643754 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.643725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkxf\" (UniqueName: \"kubernetes.io/projected/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-kube-api-access-4gkxf\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z\" (UID: \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:25.796438 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:25.796357    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:26.074283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:26.074104    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"]
Sep 16 19:07:26.077391 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:07:26.077350    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8322c9cc_edeb_43b2_bf57_9b6340dfe0c6.slice/crio-84ae8947e62169e18d448359e47973224b0f95b0efa4c84bd582691ab2656e2b WatchSource:0}: Error finding container 84ae8947e62169e18d448359e47973224b0f95b0efa4c84bd582691ab2656e2b: Status 404 returned error can't find the container with id 84ae8947e62169e18d448359e47973224b0f95b0efa4c84bd582691ab2656e2b
Sep 16 19:07:26.137828 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:26.137794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" event={"ID":"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6","Type":"ContainerStarted","Data":"84ae8947e62169e18d448359e47973224b0f95b0efa4c84bd582691ab2656e2b"}
Sep 16 19:07:27.144515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:27.144470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" event={"ID":"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6","Type":"ContainerStarted","Data":"0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed"}
Sep 16 19:07:27.144880 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:27.144623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:27.166252 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:27.166205    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" podStartSLOduration=2.166189417 podStartE2EDuration="2.166189417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:27.165152315 +0000 UTC m=+1427.426829329" watchObservedRunningTime="2026-09-16 19:07:27.166189417 +0000 UTC m=+1427.427866428"
Sep 16 19:07:28.155983 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:28.155947    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:07:38.156334 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:38.156279    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:07:48.156288 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:48.156239    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:07:58.152452 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:58.152425    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:07:58.155966 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:07:58.155941    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:08:08.156029 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:08:08.155991    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:08:18.156424 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:08:18.156385    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:08:28.156079 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:08:28.156044    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:08:38.155904 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:08:38.155846    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:08:48.165852 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:08:48.165821    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:08:48.173281 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:08:48.173258    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:08:52.497798 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:08:52.497766    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"]
Sep 16 19:09:05.392831 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:05.392741    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"]
Sep 16 19:09:07.519468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:07.519414    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" containerID="cri-o://ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23" gracePeriod=45
Sep 16 19:09:08.152918 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.152863    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:08.199020 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.198998    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:09:08.318548 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-home\") pod \"b1249f95-33b7-488c-b636-e3020a24fb16\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") "
Sep 16 19:09:08.318548 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1249f95-33b7-488c-b636-e3020a24fb16-cabundle-cert\") pod \"b1249f95-33b7-488c-b636-e3020a24fb16\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") "
Sep 16 19:09:08.318740 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-model-cache\") pod \"b1249f95-33b7-488c-b636-e3020a24fb16\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") "
Sep 16 19:09:08.318740 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318607    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-dshm\") pod \"b1249f95-33b7-488c-b636-e3020a24fb16\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") "
Sep 16 19:09:08.318740 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-tmp-dir\") pod \"b1249f95-33b7-488c-b636-e3020a24fb16\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") "
Sep 16 19:09:08.318740 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-kserve-provision-location\") pod \"b1249f95-33b7-488c-b636-e3020a24fb16\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") "
Sep 16 19:09:08.318740 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1249f95-33b7-488c-b636-e3020a24fb16-tls-certs\") pod \"b1249f95-33b7-488c-b636-e3020a24fb16\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") "
Sep 16 19:09:08.318740 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjpsr\" (UniqueName: \"kubernetes.io/projected/b1249f95-33b7-488c-b636-e3020a24fb16-kube-api-access-hjpsr\") pod \"b1249f95-33b7-488c-b636-e3020a24fb16\" (UID: \"b1249f95-33b7-488c-b636-e3020a24fb16\") "
Sep 16 19:09:08.319211 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1249f95-33b7-488c-b636-e3020a24fb16" (UID: "b1249f95-33b7-488c-b636-e3020a24fb16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:08.319211 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.318988    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1249f95-33b7-488c-b636-e3020a24fb16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1249f95-33b7-488c-b636-e3020a24fb16" (UID: "b1249f95-33b7-488c-b636-e3020a24fb16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:08.319211 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.319066    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1249f95-33b7-488c-b636-e3020a24fb16-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:08.319211 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.319085    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:08.319422 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.319313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-home" (OuterVolumeSpecName: "home") pod "b1249f95-33b7-488c-b636-e3020a24fb16" (UID: "b1249f95-33b7-488c-b636-e3020a24fb16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:08.321338 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.321312    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-dshm" (OuterVolumeSpecName: "dshm") pod "b1249f95-33b7-488c-b636-e3020a24fb16" (UID: "b1249f95-33b7-488c-b636-e3020a24fb16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:08.321773 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.321743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1249f95-33b7-488c-b636-e3020a24fb16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1249f95-33b7-488c-b636-e3020a24fb16" (UID: "b1249f95-33b7-488c-b636-e3020a24fb16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:08.321874 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.321833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1249f95-33b7-488c-b636-e3020a24fb16-kube-api-access-hjpsr" (OuterVolumeSpecName: "kube-api-access-hjpsr") pod "b1249f95-33b7-488c-b636-e3020a24fb16" (UID: "b1249f95-33b7-488c-b636-e3020a24fb16"). InnerVolumeSpecName "kube-api-access-hjpsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:08.331468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.331429    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1249f95-33b7-488c-b636-e3020a24fb16" (UID: "b1249f95-33b7-488c-b636-e3020a24fb16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:08.340507 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.340483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1249f95-33b7-488c-b636-e3020a24fb16" (UID: "b1249f95-33b7-488c-b636-e3020a24fb16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:08.419751 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.419729    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:08.419751 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.419751    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:08.419870 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.419763    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1249f95-33b7-488c-b636-e3020a24fb16-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:08.419870 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.419772    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjpsr\" (UniqueName: \"kubernetes.io/projected/b1249f95-33b7-488c-b636-e3020a24fb16-kube-api-access-hjpsr\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:08.419870 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.419779    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:08.419870 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.419787    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1249f95-33b7-488c-b636-e3020a24fb16-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:08.653054 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.652986    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1249f95-33b7-488c-b636-e3020a24fb16" containerID="ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23" exitCode=0
Sep 16 19:09:08.653054 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.653039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" event={"ID":"b1249f95-33b7-488c-b636-e3020a24fb16","Type":"ContainerDied","Data":"ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23"}
Sep 16 19:09:08.653450 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.653056    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"
Sep 16 19:09:08.653450 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.653082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" event={"ID":"b1249f95-33b7-488c-b636-e3020a24fb16","Type":"ContainerDied","Data":"9f4e89cacb6fee4b8ce0045ad91fae1ef815f474370f8f496514bc44185be2d9"}
Sep 16 19:09:08.653450 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.653104    2580 scope.go:117] "RemoveContainer" containerID="ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23"
Sep 16 19:09:08.661796 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.661781    2580 scope.go:117] "RemoveContainer" containerID="e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f"
Sep 16 19:09:08.670699 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.670682    2580 scope.go:117] "RemoveContainer" containerID="ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23"
Sep 16 19:09:08.670954 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:09:08.670935    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23\": container with ID starting with ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23 not found: ID does not exist" containerID="ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23"
Sep 16 19:09:08.671009 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.670961    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23"} err="failed to get container status \"ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23\": rpc error: code = NotFound desc = could not find container \"ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23\": container with ID starting with ffa63485c215ea6c67e2dcaba891810f8aeeadcc00869cc1df4a959b8a476f23 not found: ID does not exist"
Sep 16 19:09:08.671009 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.670977    2580 scope.go:117] "RemoveContainer" containerID="e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f"
Sep 16 19:09:08.671211 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:09:08.671192    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f\": container with ID starting with e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f not found: ID does not exist" containerID="e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f"
Sep 16 19:09:08.671266 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.671214    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f"} err="failed to get container status \"e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f\": rpc error: code = NotFound desc = could not find container \"e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f\": container with ID starting with e7a1eca6e82ff48a3341674df254380d9aed2391496fe0eb340eedc8758f323f not found: ID does not exist"
Sep 16 19:09:08.681015 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.680989    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"]
Sep 16 19:09:08.685680 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:08.685660    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k"]
Sep 16 19:09:09.168007 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:09.167966    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-64d4875f8-xrs6k" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:09:10.276035 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:10.276004    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" path="/var/lib/kubelet/pods/b1249f95-33b7-488c-b636-e3020a24fb16/volumes"
Sep 16 19:09:18.151739 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:18.151702    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:20.413596 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:20.413526    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerName="main" containerID="cri-o://0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed" gracePeriod=45
Sep 16 19:09:28.152670 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:28.152635    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:28.153137 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:28.152739    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:09:38.151632 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:38.151595    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:48.152365 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:48.152322    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:50.658202 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.658180    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:09:50.756235 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.756140    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gkxf\" (UniqueName: \"kubernetes.io/projected/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-kube-api-access-4gkxf\") pod \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\" (UID: \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\") "
Sep 16 19:09:50.756235 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.756187    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-tls-certs\") pod \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\" (UID: \"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6\") "
Sep 16 19:09:50.758517 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.758488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" (UID: "8322c9cc-edeb-43b2-bf57-9b6340dfe0c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:50.758625 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.758537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-kube-api-access-4gkxf" (OuterVolumeSpecName: "kube-api-access-4gkxf") pod "8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" (UID: "8322c9cc-edeb-43b2-bf57-9b6340dfe0c6"). InnerVolumeSpecName "kube-api-access-4gkxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:50.840809 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.840776    2580 generic.go:356] "Generic (PLEG): container finished" podID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerID="0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed" exitCode=0
Sep 16 19:09:50.841013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.840822    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"
Sep 16 19:09:50.841013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.840871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" event={"ID":"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6","Type":"ContainerDied","Data":"0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed"}
Sep 16 19:09:50.841013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.840962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z" event={"ID":"8322c9cc-edeb-43b2-bf57-9b6340dfe0c6","Type":"ContainerDied","Data":"84ae8947e62169e18d448359e47973224b0f95b0efa4c84bd582691ab2656e2b"}
Sep 16 19:09:50.841013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.840994    2580 scope.go:117] "RemoveContainer" containerID="0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed"
Sep 16 19:09:50.850015 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.849998    2580 scope.go:117] "RemoveContainer" containerID="0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed"
Sep 16 19:09:50.850249 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:09:50.850230    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed\": container with ID starting with 0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed not found: ID does not exist" containerID="0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed"
Sep 16 19:09:50.850309 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.850262    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed"} err="failed to get container status \"0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed\": rpc error: code = NotFound desc = could not find container \"0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed\": container with ID starting with 0a0168a786a907a0a86dbf5a52022956e22643d3c07a39dcf655d3332b10bbed not found: ID does not exist"
Sep 16 19:09:50.857269 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.857249    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gkxf\" (UniqueName: \"kubernetes.io/projected/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-kube-api-access-4gkxf\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:50.857344 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.857272    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:09:50.861101 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.861083    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"]
Sep 16 19:09:50.863225 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:50.863208    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75k9c7z"]
Sep 16 19:09:52.275344 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:09:52.275306    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" path="/var/lib/kubelet/pods/8322c9cc-edeb-43b2-bf57-9b6340dfe0c6/volumes"
Sep 16 19:10:04.812709 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.812673    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"]
Sep 16 19:10:04.815081 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.813030    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerName="main"
Sep 16 19:10:04.815081 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.813042    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerName="main"
Sep 16 19:10:04.815081 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.813054    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main"
Sep 16 19:10:04.815081 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.813059    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main"
Sep 16 19:10:04.815081 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.813071    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="storage-initializer"
Sep 16 19:10:04.815081 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.813078    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="storage-initializer"
Sep 16 19:10:04.815081 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.813134    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1249f95-33b7-488c-b636-e3020a24fb16" containerName="main"
Sep 16 19:10:04.815081 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.813143    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8322c9cc-edeb-43b2-bf57-9b6340dfe0c6" containerName="main"
Sep 16 19:10:04.815989 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.815975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.819381 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.819358    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 16 19:10:04.820136 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.820120    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 19:10:04.820247 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.820231    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 19:10:04.821144 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.821130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-zhwz6\""
Sep 16 19:10:04.829713 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.829693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"]
Sep 16 19:10:04.885261 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.885236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-dshm\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.885261 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.885266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.885440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.885291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896cn\" (UniqueName: \"kubernetes.io/projected/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-kube-api-access-896cn\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.885440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.885346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.885440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.885364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-home\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.885440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.885427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.986849 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.986820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.987027 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.986904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-dshm\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.987027 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.986938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.987027 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.986960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896cn\" (UniqueName: \"kubernetes.io/projected/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-kube-api-access-896cn\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.987027 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.986984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.987271 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.987035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-home\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.987271 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.987218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.987376 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.987353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-home\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.987453 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.987436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.989258 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.989238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-dshm\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:04.989577 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:04.989561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:05.004087 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:05.004066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-896cn\" (UniqueName: \"kubernetes.io/projected/8cc70eba-b97c-41c6-a169-5fbdd8529bfa-kube-api-access-896cn\") pod \"canary-v2-kserve-7f65dbbccc-2s557\" (UID: \"8cc70eba-b97c-41c6-a169-5fbdd8529bfa\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:05.125586 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:05.125499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:05.403240 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:05.403213    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"]
Sep 16 19:10:05.405774 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:10:05.405752    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc70eba_b97c_41c6_a169_5fbdd8529bfa.slice/crio-cdf719e52352e3eacd184844f23d6f3e0677150cfc901ed7327a1c37e7726b05 WatchSource:0}: Error finding container cdf719e52352e3eacd184844f23d6f3e0677150cfc901ed7327a1c37e7726b05: Status 404 returned error can't find the container with id cdf719e52352e3eacd184844f23d6f3e0677150cfc901ed7327a1c37e7726b05
Sep 16 19:10:05.407623 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:05.407607    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:10:05.912702 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:05.912667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557" event={"ID":"8cc70eba-b97c-41c6-a169-5fbdd8529bfa","Type":"ContainerStarted","Data":"cdf719e52352e3eacd184844f23d6f3e0677150cfc901ed7327a1c37e7726b05"}
Sep 16 19:10:06.919409 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:06.919375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557" event={"ID":"8cc70eba-b97c-41c6-a169-5fbdd8529bfa","Type":"ContainerStarted","Data":"08df78549e93f2342d1757afedb02a3b9ae55ff8281acad42e40555166209fa9"}
Sep 16 19:10:06.937005 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:06.936955    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557" podStartSLOduration=1.742422734 podStartE2EDuration="2.936934773s" totalImagesPullingTime="1.194512039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:06.935910531 +0000 UTC m=+1587.197587533" watchObservedRunningTime="2026-09-16 19:10:06.936934773 +0000 UTC m=+1587.198611785"
Sep 16 19:10:07.126440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:07.126402    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:15.125830 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:15.125796    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:15.138423 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:15.138399    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:15.972532 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:15.972504    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-2s557"
Sep 16 19:10:31.360420 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.360338    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"]
Sep 16 19:10:31.363761 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.363742    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:10:31.367974 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.367952    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:10:31.368098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.367977    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-ldrjs\""
Sep 16 19:10:31.368098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.367995    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:10:31.368098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.367977    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-sjzrv\""
Sep 16 19:10:31.368098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.368025    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:10:31.379267 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.379244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"]
Sep 16 19:10:31.515107 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.515067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a66818-1ef4-4524-aecf-b22b82bfa90e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v\" (UID: \"44a66818-1ef4-4524-aecf-b22b82bfa90e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:10:31.515277 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.515140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbf2r\" (UniqueName: \"kubernetes.io/projected/44a66818-1ef4-4524-aecf-b22b82bfa90e-kube-api-access-pbf2r\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v\" (UID: \"44a66818-1ef4-4524-aecf-b22b82bfa90e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:10:31.615673 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.615581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a66818-1ef4-4524-aecf-b22b82bfa90e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v\" (UID: \"44a66818-1ef4-4524-aecf-b22b82bfa90e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:10:31.615673 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.615647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbf2r\" (UniqueName: \"kubernetes.io/projected/44a66818-1ef4-4524-aecf-b22b82bfa90e-kube-api-access-pbf2r\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v\" (UID: \"44a66818-1ef4-4524-aecf-b22b82bfa90e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:10:31.618587 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.618559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a66818-1ef4-4524-aecf-b22b82bfa90e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v\" (UID: \"44a66818-1ef4-4524-aecf-b22b82bfa90e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:10:31.623724 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.623705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbf2r\" (UniqueName: \"kubernetes.io/projected/44a66818-1ef4-4524-aecf-b22b82bfa90e-kube-api-access-pbf2r\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v\" (UID: \"44a66818-1ef4-4524-aecf-b22b82bfa90e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:10:31.674403 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.674375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:10:31.971454 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:31.971411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"]
Sep 16 19:10:31.974325 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:10:31.974293    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a66818_1ef4_4524_aecf_b22b82bfa90e.slice/crio-6ce29fb0b63972020bae4f1c8b4881e5c8c242264757fd5d615a0938970f820f WatchSource:0}: Error finding container 6ce29fb0b63972020bae4f1c8b4881e5c8c242264757fd5d615a0938970f820f: Status 404 returned error can't find the container with id 6ce29fb0b63972020bae4f1c8b4881e5c8c242264757fd5d615a0938970f820f
Sep 16 19:10:32.045766 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:32.045724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" event={"ID":"44a66818-1ef4-4524-aecf-b22b82bfa90e","Type":"ContainerStarted","Data":"a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f"}
Sep 16 19:10:32.045957 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:32.045882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" event={"ID":"44a66818-1ef4-4524-aecf-b22b82bfa90e","Type":"ContainerStarted","Data":"6ce29fb0b63972020bae4f1c8b4881e5c8c242264757fd5d615a0938970f820f"}
Sep 16 19:10:32.046032 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:32.045968    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:10:32.070464 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:10:32.070406    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" podStartSLOduration=1.070383467 podStartE2EDuration="1.070383467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:32.068283926 +0000 UTC m=+1612.329960959" watchObservedRunningTime="2026-09-16 19:10:32.070383467 +0000 UTC m=+1612.332060479"
Sep 16 19:11:03.052710 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:11:03.052681    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:11:25.510699 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:11:25.510664    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"]
Sep 16 19:11:33.053047 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:11:33.053005    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" podUID="44a66818-1ef4-4524-aecf-b22b82bfa90e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:40.531430 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:11:40.531358    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" podUID="44a66818-1ef4-4524-aecf-b22b82bfa90e" containerName="main" containerID="cri-o://a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f" gracePeriod=45
Sep 16 19:11:43.052206 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:11:43.052168    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" podUID="44a66818-1ef4-4524-aecf-b22b82bfa90e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:53.052382 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:11:53.052339    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" podUID="44a66818-1ef4-4524-aecf-b22b82bfa90e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:53.052813 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:11:53.052475    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:12:03.052259 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:03.052176    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" podUID="44a66818-1ef4-4524-aecf-b22b82bfa90e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:12:10.788169 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:10.788133    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:12:10.807104 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:10.806848    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbf2r\" (UniqueName: \"kubernetes.io/projected/44a66818-1ef4-4524-aecf-b22b82bfa90e-kube-api-access-pbf2r\") pod \"44a66818-1ef4-4524-aecf-b22b82bfa90e\" (UID: \"44a66818-1ef4-4524-aecf-b22b82bfa90e\") "
Sep 16 19:12:10.807104 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:10.807043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a66818-1ef4-4524-aecf-b22b82bfa90e-tls-certs\") pod \"44a66818-1ef4-4524-aecf-b22b82bfa90e\" (UID: \"44a66818-1ef4-4524-aecf-b22b82bfa90e\") "
Sep 16 19:12:10.810155 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:10.810112    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a66818-1ef4-4524-aecf-b22b82bfa90e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44a66818-1ef4-4524-aecf-b22b82bfa90e" (UID: "44a66818-1ef4-4524-aecf-b22b82bfa90e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:10.810300 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:10.810200    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a66818-1ef4-4524-aecf-b22b82bfa90e-kube-api-access-pbf2r" (OuterVolumeSpecName: "kube-api-access-pbf2r") pod "44a66818-1ef4-4524-aecf-b22b82bfa90e" (UID: "44a66818-1ef4-4524-aecf-b22b82bfa90e"). InnerVolumeSpecName "kube-api-access-pbf2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:10.908033 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:10.907991    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbf2r\" (UniqueName: \"kubernetes.io/projected/44a66818-1ef4-4524-aecf-b22b82bfa90e-kube-api-access-pbf2r\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:12:10.908033 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:10.908028    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a66818-1ef4-4524-aecf-b22b82bfa90e-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:12:11.520531 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:11.520496    2580 generic.go:356] "Generic (PLEG): container finished" podID="44a66818-1ef4-4524-aecf-b22b82bfa90e" containerID="a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f" exitCode=0
Sep 16 19:12:11.520840 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:11.520548    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"
Sep 16 19:12:11.520840 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:11.520580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" event={"ID":"44a66818-1ef4-4524-aecf-b22b82bfa90e","Type":"ContainerDied","Data":"a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f"}
Sep 16 19:12:11.520840 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:11.520637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v" event={"ID":"44a66818-1ef4-4524-aecf-b22b82bfa90e","Type":"ContainerDied","Data":"6ce29fb0b63972020bae4f1c8b4881e5c8c242264757fd5d615a0938970f820f"}
Sep 16 19:12:11.520840 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:11.520660    2580 scope.go:117] "RemoveContainer" containerID="a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f"
Sep 16 19:12:11.530751 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:11.530733    2580 scope.go:117] "RemoveContainer" containerID="a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f"
Sep 16 19:12:11.531134 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:11.531112    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f\": container with ID starting with a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f not found: ID does not exist" containerID="a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f"
Sep 16 19:12:11.531194 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:11.531144    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f"} err="failed to get container status \"a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f\": rpc error: code = NotFound desc = could not find container \"a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f\": container with ID starting with a0c567a81cf15326699020c8eb0b260a732c86b650e9d668bf54d75391626a0f not found: ID does not exist"
Sep 16 19:12:11.545230 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:11.545195    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"]
Sep 16 19:12:11.553270 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:11.553237    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2lk7v"]
Sep 16 19:12:12.277128 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:12.277098    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a66818-1ef4-4524-aecf-b22b82bfa90e" path="/var/lib/kubelet/pods/44a66818-1ef4-4524-aecf-b22b82bfa90e/volumes"
Sep 16 19:12:26.560383 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.560346    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"]
Sep 16 19:12:26.560885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.560783    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a66818-1ef4-4524-aecf-b22b82bfa90e" containerName="main"
Sep 16 19:12:26.560885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.560794    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a66818-1ef4-4524-aecf-b22b82bfa90e" containerName="main"
Sep 16 19:12:26.560885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.560867    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a66818-1ef4-4524-aecf-b22b82bfa90e" containerName="main"
Sep 16 19:12:26.565298 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.565277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.567976 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:26.567941    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"seaweedfs-s3-creds\" is forbidden: User \"system:node:ip-10-0-139-95.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-cf47448b\": no relationship found between node 'ip-10-0-139-95.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\"" type="*v1.Secret"
Sep 16 19:12:26.568488 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:26.568449    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: User \"system:node:ip-10-0-139-95.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-cf47448b\": no relationship found between node 'ip-10-0-139-95.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\"" type="*v1.ConfigMap"
Sep 16 19:12:26.568953 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:26.568917    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-95.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-cf47448b\": no relationship found between node 'ip-10-0-139-95.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 19:12:26.569106 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:26.569085    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-95.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-cf47448b\": no relationship found between node 'ip-10-0-139-95.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 19:12:26.569352 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:26.569323    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" is forbidden: User \"system:node:ip-10-0-139-95.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-cf47448b\": no relationship found between node 'ip-10-0-139-95.ec2.internal' and this object" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.569658 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:26.569632    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"tokenizer-clean-path-test-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-139-95.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-cf47448b\": no relationship found between node 'ip-10-0-139-95.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 16 19:12:26.569701 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:26.569663    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-tzf24\" is forbidden: User \"system:node:ip-10-0-139-95.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-cf47448b\": no relationship found between node 'ip-10-0-139-95.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-tzf24\"" type="*v1.Secret"
Sep 16 19:12:26.588959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.588920    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"]
Sep 16 19:12:26.652664 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.652585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba671df4-5c9c-4497-a8ab-9754599a18a1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.652664 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.652655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-model-cache\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.653163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.652696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba671df4-5c9c-4497-a8ab-9754599a18a1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.653163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.652841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.653163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.653000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-dshm\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.653163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.653026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.653163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.653071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-home\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.653348 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.653206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdf8\" (UniqueName: \"kubernetes.io/projected/ba671df4-5c9c-4497-a8ab-9754599a18a1-kube-api-access-bfdf8\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.753809 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.753770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-home\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.753836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdf8\" (UniqueName: \"kubernetes.io/projected/ba671df4-5c9c-4497-a8ab-9754599a18a1-kube-api-access-bfdf8\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.753887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba671df4-5c9c-4497-a8ab-9754599a18a1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.753951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-model-cache\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.753977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba671df4-5c9c-4497-a8ab-9754599a18a1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754233 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.754039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754233 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.754083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-dshm\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754233 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.754149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754393 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.754299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-home\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754393 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.754306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-model-cache\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754393 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.754382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.754551 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.754486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.756374 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.756357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-dshm\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:26.967278 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.967245    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"]
Sep 16 19:12:26.970939 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:26.970915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.004719 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.004687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"]
Sep 16 19:12:27.057886 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.057851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.057886 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.057918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.057886 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.057949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.058212 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.058033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.158549 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.158482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.158549 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.158541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.158784 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.158569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.158784 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.158675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.159031 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.159015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.159088 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.159047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.381536 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.381447    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:12:27.386866 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.386832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba671df4-5c9c-4497-a8ab-9754599a18a1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:27.392010 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.391979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.506687 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.506651    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:12:27.541431 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.541394    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:12:27.544807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.544782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdf8\" (UniqueName: \"kubernetes.io/projected/ba671df4-5c9c-4497-a8ab-9754599a18a1-kube-api-access-bfdf8\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:27.754784 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:27.754749    2580 configmap.go:193] Couldn't get configMap e2e-test-llm-inference-service-cf47448b/odh-kserve-custom-ca-bundle: failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:12:27.755196 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:12:27.754848    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba671df4-5c9c-4497-a8ab-9754599a18a1-cabundle-cert podName:ba671df4-5c9c-4497-a8ab-9754599a18a1 nodeName:}" failed. No retries permitted until 2026-09-16 19:12:28.254831853 +0000 UTC m=+1728.516508842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cabundle-cert" (UniqueName: "kubernetes.io/configmap/ba671df4-5c9c-4497-a8ab-9754599a18a1-cabundle-cert") pod "tokenizer-clean-path-test-kserve-fc98d4fc-24srd" (UID: "ba671df4-5c9c-4497-a8ab-9754599a18a1") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:12:27.810198 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.810163    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:27.889242 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.889204    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:27.899932 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.899871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:27.995388 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:27.995351    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-tzf24\""
Sep 16 19:12:28.184467 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:28.184429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:12:28.268749 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:28.268697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba671df4-5c9c-4497-a8ab-9754599a18a1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:28.269555 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:28.269531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba671df4-5c9c-4497-a8ab-9754599a18a1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-fc98d4fc-24srd\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:28.375526 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:28.375485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:28.474226 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:28.474195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"]
Sep 16 19:12:28.476436 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:12:28.476397    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f7b210_1e6a_4e15_96b9_ca7f0ee1914e.slice/crio-f1f1c2595c625ac0105b2527c51489212d27aa1606cc4dc347835efe247bf822 WatchSource:0}: Error finding container f1f1c2595c625ac0105b2527c51489212d27aa1606cc4dc347835efe247bf822: Status 404 returned error can't find the container with id f1f1c2595c625ac0105b2527c51489212d27aa1606cc4dc347835efe247bf822
Sep 16 19:12:28.606579 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:28.606539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7" event={"ID":"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e","Type":"ContainerStarted","Data":"82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423"}
Sep 16 19:12:28.606775 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:28.606601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7" event={"ID":"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e","Type":"ContainerStarted","Data":"f1f1c2595c625ac0105b2527c51489212d27aa1606cc4dc347835efe247bf822"}
Sep 16 19:12:28.670682 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:28.670653    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"]
Sep 16 19:12:28.673262 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:12:28.673226    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba671df4_5c9c_4497_a8ab_9754599a18a1.slice/crio-aaa2c43e247c64e0821e6886f2265f95d608ec17e35b85cebb5ae793973541b9 WatchSource:0}: Error finding container aaa2c43e247c64e0821e6886f2265f95d608ec17e35b85cebb5ae793973541b9: Status 404 returned error can't find the container with id aaa2c43e247c64e0821e6886f2265f95d608ec17e35b85cebb5ae793973541b9
Sep 16 19:12:29.614521 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:29.614482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" event={"ID":"ba671df4-5c9c-4497-a8ab-9754599a18a1","Type":"ContainerStarted","Data":"35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9"}
Sep 16 19:12:29.614983 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:29.614552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" event={"ID":"ba671df4-5c9c-4497-a8ab-9754599a18a1","Type":"ContainerStarted","Data":"aaa2c43e247c64e0821e6886f2265f95d608ec17e35b85cebb5ae793973541b9"}
Sep 16 19:12:31.626268 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:31.626181    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerID="35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9" exitCode=0
Sep 16 19:12:31.626268 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:31.626255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" event={"ID":"ba671df4-5c9c-4497-a8ab-9754599a18a1","Type":"ContainerDied","Data":"35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9"}
Sep 16 19:12:31.627993 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:31.627962    2580 generic.go:356] "Generic (PLEG): container finished" podID="f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" containerID="82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423" exitCode=0
Sep 16 19:12:31.628107 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:31.628018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7" event={"ID":"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e","Type":"ContainerDied","Data":"82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423"}
Sep 16 19:12:32.635387 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:32.635351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" event={"ID":"ba671df4-5c9c-4497-a8ab-9754599a18a1","Type":"ContainerStarted","Data":"c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab"}
Sep 16 19:12:32.688976 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:32.688915    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" podStartSLOduration=4.015313579 podStartE2EDuration="6.688878474s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.673564895s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:32.688016481 +0000 UTC m=+1732.949693505" watchObservedRunningTime="2026-09-16 19:12:32.688878474 +0000 UTC m=+1732.950555486"
Sep 16 19:12:33.376013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:33.375975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:38.376077 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:38.375771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:38.392469 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:38.392441    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:38.690413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:38.690179    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:12:50.782021 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:12:50.781982    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-canary-service-backend-7b63470c_canary-v2-kserve-7f65dbbccc-2s557_8cc70eba-b97c-41c6-a169-5fbdd8529bfa/main/0.log"
Sep 16 19:13:19.909456 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:13:19.909422    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7" event={"ID":"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e","Type":"ContainerStarted","Data":"6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1"}
Sep 16 19:13:19.909876 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:13:19.909643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:13:19.930504 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:13:19.930453    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7" podStartSLOduration=3.072835331 podStartE2EDuration="53.930437839s" totalImagesPullingTime="48.126166354s" totalInitContainerRuntime="2.731436154s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:19.929389725 +0000 UTC m=+1780.191066738" watchObservedRunningTime="2026-09-16 19:13:19.930437839 +0000 UTC m=+1780.192114851"
Sep 16 19:13:50.926416 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:13:50.926386    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:14:23.632445 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:23.632409    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7_f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e/main/0.log"
Sep 16 19:14:47.427331 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.427297    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"]
Sep 16 19:14:47.427796 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.427530    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7" podUID="f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" containerName="main" containerID="cri-o://6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1" gracePeriod=30
Sep 16 19:14:47.440310 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.440277    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"]
Sep 16 19:14:47.450759 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:14:47.450719    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:14:47.450759 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:47.450759 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:47.450759 ip-10-0-139-95 kubenswrapper[2580]:  > containerID="c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab" cmd=["/bin/sleep","15"]
Sep 16 19:14:47.451112 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.450784    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" message="" err=<
Sep 16 19:14:47.451112 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:47.451112 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:47.451112 ip-10-0-139-95 kubenswrapper[2580]:  >
Sep 16 19:14:47.451112 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:14:47.450802    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:14:47.451112 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:47.451112 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:47.451112 ip-10-0-139-95 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerName="main" containerID="cri-o://c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab"
Sep 16 19:14:47.451112 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.450881    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerName="main" containerID="cri-o://c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab" gracePeriod=60
Sep 16 19:14:47.674463 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.674420    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 19:14:47.812047 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.812023    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:14:47.887261 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-kserve-provision-location\") pod \"ba671df4-5c9c-4497-a8ab-9754599a18a1\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") "
Sep 16 19:14:47.887468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-model-cache\") pod \"ba671df4-5c9c-4497-a8ab-9754599a18a1\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") "
Sep 16 19:14:47.887468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba671df4-5c9c-4497-a8ab-9754599a18a1-cabundle-cert\") pod \"ba671df4-5c9c-4497-a8ab-9754599a18a1\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") "
Sep 16 19:14:47.887468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-tmp-dir\") pod \"ba671df4-5c9c-4497-a8ab-9754599a18a1\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") "
Sep 16 19:14:47.887468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfdf8\" (UniqueName: \"kubernetes.io/projected/ba671df4-5c9c-4497-a8ab-9754599a18a1-kube-api-access-bfdf8\") pod \"ba671df4-5c9c-4497-a8ab-9754599a18a1\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") "
Sep 16 19:14:47.887468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887395    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-dshm\") pod \"ba671df4-5c9c-4497-a8ab-9754599a18a1\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") "
Sep 16 19:14:47.887468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba671df4-5c9c-4497-a8ab-9754599a18a1-tls-certs\") pod \"ba671df4-5c9c-4497-a8ab-9754599a18a1\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") "
Sep 16 19:14:47.887468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887449    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-home\") pod \"ba671df4-5c9c-4497-a8ab-9754599a18a1\" (UID: \"ba671df4-5c9c-4497-a8ab-9754599a18a1\") "
Sep 16 19:14:47.887875 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887653    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba671df4-5c9c-4497-a8ab-9754599a18a1" (UID: "ba671df4-5c9c-4497-a8ab-9754599a18a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:47.887875 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-home" (OuterVolumeSpecName: "home") pod "ba671df4-5c9c-4497-a8ab-9754599a18a1" (UID: "ba671df4-5c9c-4497-a8ab-9754599a18a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:47.888010 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.887980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba671df4-5c9c-4497-a8ab-9754599a18a1" (UID: "ba671df4-5c9c-4497-a8ab-9754599a18a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:47.888133 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.888097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba671df4-5c9c-4497-a8ab-9754599a18a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba671df4-5c9c-4497-a8ab-9754599a18a1" (UID: "ba671df4-5c9c-4497-a8ab-9754599a18a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:47.890082 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.890045    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-dshm" (OuterVolumeSpecName: "dshm") pod "ba671df4-5c9c-4497-a8ab-9754599a18a1" (UID: "ba671df4-5c9c-4497-a8ab-9754599a18a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:47.890208 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.890071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba671df4-5c9c-4497-a8ab-9754599a18a1-kube-api-access-bfdf8" (OuterVolumeSpecName: "kube-api-access-bfdf8") pod "ba671df4-5c9c-4497-a8ab-9754599a18a1" (UID: "ba671df4-5c9c-4497-a8ab-9754599a18a1"). InnerVolumeSpecName "kube-api-access-bfdf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:47.890208 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.890113    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba671df4-5c9c-4497-a8ab-9754599a18a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba671df4-5c9c-4497-a8ab-9754599a18a1" (UID: "ba671df4-5c9c-4497-a8ab-9754599a18a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:47.920283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.920227    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba671df4-5c9c-4497-a8ab-9754599a18a1" (UID: "ba671df4-5c9c-4497-a8ab-9754599a18a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:47.988261 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.988232    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba671df4-5c9c-4497-a8ab-9754599a18a1-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.988261 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.988260    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.988427 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.988269    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfdf8\" (UniqueName: \"kubernetes.io/projected/ba671df4-5c9c-4497-a8ab-9754599a18a1-kube-api-access-bfdf8\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.988427 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.988280    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.988427 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.988289    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba671df4-5c9c-4497-a8ab-9754599a18a1-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.988427 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.988297    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.988427 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.988305    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.988427 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:47.988313    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba671df4-5c9c-4497-a8ab-9754599a18a1-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.104301 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.104277    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:14:48.189732 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.189692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tmp-dir\") pod \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") "
Sep 16 19:14:48.189962 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.189758    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-kserve-provision-location\") pod \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") "
Sep 16 19:14:48.189962 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.189784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tls-certs\") pod \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") "
Sep 16 19:14:48.189962 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.189824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-cabundle-cert\") pod \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\" (UID: \"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e\") "
Sep 16 19:14:48.190250 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.190223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" (UID: "f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:48.190250 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.190237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" (UID: "f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:48.192102 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.192083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" (UID: "f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:48.210982 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.210940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" (UID: "f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:48.290639 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.290402    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.290639 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.290442    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.290639 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.290452    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.290639 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.290462    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:14:48.344413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.344366    2580 generic.go:356] "Generic (PLEG): container finished" podID="f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" containerID="6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1" exitCode=0
Sep 16 19:14:48.344608 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.344449    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"
Sep 16 19:14:48.344608 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.344449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7" event={"ID":"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e","Type":"ContainerDied","Data":"6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1"}
Sep 16 19:14:48.344608 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.344511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7" event={"ID":"f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e","Type":"ContainerDied","Data":"f1f1c2595c625ac0105b2527c51489212d27aa1606cc4dc347835efe247bf822"}
Sep 16 19:14:48.344608 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.344543    2580 scope.go:117] "RemoveContainer" containerID="6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1"
Sep 16 19:14:48.346603 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.346572    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerID="c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab" exitCode=0
Sep 16 19:14:48.346772 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.346654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" event={"ID":"ba671df4-5c9c-4497-a8ab-9754599a18a1","Type":"ContainerDied","Data":"c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab"}
Sep 16 19:14:48.346772 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.346679    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"
Sep 16 19:14:48.346772 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.346706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd" event={"ID":"ba671df4-5c9c-4497-a8ab-9754599a18a1","Type":"ContainerDied","Data":"aaa2c43e247c64e0821e6886f2265f95d608ec17e35b85cebb5ae793973541b9"}
Sep 16 19:14:48.353425 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.353404    2580 scope.go:117] "RemoveContainer" containerID="82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423"
Sep 16 19:14:48.361413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.361384    2580 scope.go:117] "RemoveContainer" containerID="6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1"
Sep 16 19:14:48.361796 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:14:48.361762    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1\": container with ID starting with 6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1 not found: ID does not exist" containerID="6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1"
Sep 16 19:14:48.361870 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.361794    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1"} err="failed to get container status \"6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1\": rpc error: code = NotFound desc = could not find container \"6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1\": container with ID starting with 6cceded312725520672f32edac8ab2851a8d6d9d404c5abed65b5d4829f092c1 not found: ID does not exist"
Sep 16 19:14:48.361870 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.361812    2580 scope.go:117] "RemoveContainer" containerID="82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423"
Sep 16 19:14:48.362147 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:14:48.362127    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423\": container with ID starting with 82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423 not found: ID does not exist" containerID="82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423"
Sep 16 19:14:48.362223 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.362157    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423"} err="failed to get container status \"82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423\": rpc error: code = NotFound desc = could not find container \"82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423\": container with ID starting with 82a8b348f8b9218fb9d114d39ab891aa1f400c8bd4756aef66bd2c0f67e0e423 not found: ID does not exist"
Sep 16 19:14:48.362223 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.362179    2580 scope.go:117] "RemoveContainer" containerID="c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab"
Sep 16 19:14:48.363175 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.363116    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"]
Sep 16 19:14:48.370911 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.370870    2580 scope.go:117] "RemoveContainer" containerID="35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9"
Sep 16 19:14:48.371650 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.371631    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-fc98d4fc-24srd"]
Sep 16 19:14:48.379158 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.379140    2580 scope.go:117] "RemoveContainer" containerID="c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab"
Sep 16 19:14:48.379445 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:14:48.379428    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab\": container with ID starting with c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab not found: ID does not exist" containerID="c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab"
Sep 16 19:14:48.379493 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.379453    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab"} err="failed to get container status \"c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab\": rpc error: code = NotFound desc = could not find container \"c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab\": container with ID starting with c2299f3a98db7ad294f3e691fb70306b78446a3781ae35f2d109010d212c52ab not found: ID does not exist"
Sep 16 19:14:48.379493 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.379470    2580 scope.go:117] "RemoveContainer" containerID="35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9"
Sep 16 19:14:48.379684 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:14:48.379669    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9\": container with ID starting with 35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9 not found: ID does not exist" containerID="35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9"
Sep 16 19:14:48.379723 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.379689    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9"} err="failed to get container status \"35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9\": rpc error: code = NotFound desc = could not find container \"35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9\": container with ID starting with 35b5f8b749c632a5f9dfcc316553af3da480c727c94feaec2afca22f8a536ac9 not found: ID does not exist"
Sep 16 19:14:48.386957 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.386918    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"]
Sep 16 19:14:48.391565 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:48.391542    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-65b864944d-rgxw7"]
Sep 16 19:14:50.276079 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:50.276043    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" path="/var/lib/kubelet/pods/ba671df4-5c9c-4497-a8ab-9754599a18a1/volumes"
Sep 16 19:14:50.276502 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:14:50.276488    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" path="/var/lib/kubelet/pods/f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e/volumes"
Sep 16 19:15:31.114736 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.114691    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"]
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115190    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" containerName="storage-initializer"
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115210    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" containerName="storage-initializer"
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115254    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerName="main"
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115264    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerName="main"
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115279    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerName="storage-initializer"
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115289    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerName="storage-initializer"
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115299    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" containerName="main"
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115308    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" containerName="main"
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115388    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba671df4-5c9c-4497-a8ab-9754599a18a1" containerName="main"
Sep 16 19:15:31.116260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.115397    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4f7b210-1e6a-4e15-96b9-ca7f0ee1914e" containerName="main"
Sep 16 19:15:31.117245 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.117225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.119625 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.119601    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:15:31.119778 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.119756    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-9b9gb\""
Sep 16 19:15:31.120512 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.120488    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 16 19:15:31.120612 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.120542    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:15:31.130318 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.130292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"]
Sep 16 19:15:31.244984 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.244949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-model-cache\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.244984 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.244982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.245187 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.245013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-home\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.245187 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.245125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4vb\" (UniqueName: \"kubernetes.io/projected/1f95169d-db02-47bd-8ab5-b37290b51441-kube-api-access-fv4vb\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.245187 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.245170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-dshm\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.245294 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.245221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f95169d-db02-47bd-8ab5-b37290b51441-tls-certs\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.346289 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.346258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-dshm\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.346466 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.346314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f95169d-db02-47bd-8ab5-b37290b51441-tls-certs\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.346466 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.346367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-model-cache\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.346466 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.346388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.346466 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.346429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-home\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.346681 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.346493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4vb\" (UniqueName: \"kubernetes.io/projected/1f95169d-db02-47bd-8ab5-b37290b51441-kube-api-access-fv4vb\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.346777 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.346754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-model-cache\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.346846 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.346796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.346846 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.346820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-home\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.348553 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.348537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-dshm\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.348795 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.348778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f95169d-db02-47bd-8ab5-b37290b51441-tls-certs\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.355234 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.355215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4vb\" (UniqueName: \"kubernetes.io/projected/1f95169d-db02-47bd-8ab5-b37290b51441-kube-api-access-fv4vb\") pod \"stop-v1-kserve-5657b88c95-wwnzk\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.430217 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.430191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:31.713733 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:31.713705    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"]
Sep 16 19:15:31.716510 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:15:31.716480    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f95169d_db02_47bd_8ab5_b37290b51441.slice/crio-b8399b19a960ae5d68f80f36d7f2bea8ad81db65864171d2fd57923efb261a7e WatchSource:0}: Error finding container b8399b19a960ae5d68f80f36d7f2bea8ad81db65864171d2fd57923efb261a7e: Status 404 returned error can't find the container with id b8399b19a960ae5d68f80f36d7f2bea8ad81db65864171d2fd57923efb261a7e
Sep 16 19:15:32.550611 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:32.550571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk" event={"ID":"1f95169d-db02-47bd-8ab5-b37290b51441","Type":"ContainerStarted","Data":"f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971"}
Sep 16 19:15:32.551062 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:32.550623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk" event={"ID":"1f95169d-db02-47bd-8ab5-b37290b51441","Type":"ContainerStarted","Data":"b8399b19a960ae5d68f80f36d7f2bea8ad81db65864171d2fd57923efb261a7e"}
Sep 16 19:15:32.574945 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:32.570165    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk" podStartSLOduration=1.570144365 podStartE2EDuration="1.570144365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:32.568500327 +0000 UTC m=+1912.830177351" watchObservedRunningTime="2026-09-16 19:15:32.570144365 +0000 UTC m=+1912.831821377"
Sep 16 19:15:33.430985 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:33.430952    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:41.430392 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:41.430345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:41.443048 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:41.443022    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:15:41.600999 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:15:41.600971    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:16:18.623083 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.623052    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"]
Sep 16 19:16:18.625610 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.625594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.629093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.629070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:16:18.629093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.629084    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:16:18.629288 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.629147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:16:18.629691 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.629672    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:16:18.629814 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.629721    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:16:18.629814 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.629776    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-85x4z\""
Sep 16 19:16:18.660707 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.660676    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"]
Sep 16 19:16:18.762286 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.762250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-home\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.762460 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.762310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.762460 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.762374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tls-certs\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.762460 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.762394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-dshm\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.762460 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.762425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.762618 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.762462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.762618 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.762574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-model-cache\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.762686 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.762614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qp6h\" (UniqueName: \"kubernetes.io/projected/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kube-api-access-5qp6h\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863300 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863450 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863450 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-model-cache\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863560 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qp6h\" (UniqueName: \"kubernetes.io/projected/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kube-api-access-5qp6h\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863560 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-home\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863560 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863718 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tls-certs\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863718 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-dshm\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863718 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-model-cache\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.863972 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.864051 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.863973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-home\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.864098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.864064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.864098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.864074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.866049 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.866032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-dshm\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.866148 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.866133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tls-certs\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.870971 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.870947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qp6h\" (UniqueName: \"kubernetes.io/projected/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kube-api-access-5qp6h\") pod \"tokenizer-migration-path-test-kserve-774757bb8d-8fgrx\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.924677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.924651    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"]
Sep 16 19:16:18.927415 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.927397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:18.935466 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.935447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:18.944753 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:18.944730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"]
Sep 16 19:16:19.066135 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.066100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.066135 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.066135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acfc4afb-8b3f-4337-b730-6d0f542b0472-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.066345 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.066155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.066345 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.066180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acfc4afb-8b3f-4337-b730-6d0f542b0472-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.166757 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.166726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.166757 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.166757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acfc4afb-8b3f-4337-b730-6d0f542b0472-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.167013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.166776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.167013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.166792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acfc4afb-8b3f-4337-b730-6d0f542b0472-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.167162 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.167144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.167225 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.167175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.167446 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.167421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acfc4afb-8b3f-4337-b730-6d0f542b0472-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.169540 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.169519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acfc4afb-8b3f-4337-b730-6d0f542b0472-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.215195 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.215176    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"]
Sep 16 19:16:19.217429 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:16:19.217401    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d01764a_9d72_48ed_8ab7_d7db4a4405ef.slice/crio-6452056f902c7531c3b505218385f332ebafb04026633611ac7d324927f6030c WatchSource:0}: Error finding container 6452056f902c7531c3b505218385f332ebafb04026633611ac7d324927f6030c: Status 404 returned error can't find the container with id 6452056f902c7531c3b505218385f332ebafb04026633611ac7d324927f6030c
Sep 16 19:16:19.239143 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.239121    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:19.509124 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.509100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"]
Sep 16 19:16:19.511363 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:16:19.511337    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfc4afb_8b3f_4337_b730_6d0f542b0472.slice/crio-5a53d25cffd9a41332cb5d4f0e1c523a75a41315b9c28eace5e4b3e7f51ede92 WatchSource:0}: Error finding container 5a53d25cffd9a41332cb5d4f0e1c523a75a41315b9c28eace5e4b3e7f51ede92: Status 404 returned error can't find the container with id 5a53d25cffd9a41332cb5d4f0e1c523a75a41315b9c28eace5e4b3e7f51ede92
Sep 16 19:16:19.773686 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.773526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" event={"ID":"1d01764a-9d72-48ed-8ab7-d7db4a4405ef","Type":"ContainerStarted","Data":"7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd"}
Sep 16 19:16:19.773686 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.773631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" event={"ID":"1d01764a-9d72-48ed-8ab7-d7db4a4405ef","Type":"ContainerStarted","Data":"6452056f902c7531c3b505218385f332ebafb04026633611ac7d324927f6030c"}
Sep 16 19:16:19.775466 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.775437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4" event={"ID":"acfc4afb-8b3f-4337-b730-6d0f542b0472","Type":"ContainerStarted","Data":"86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52"}
Sep 16 19:16:19.775614 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:19.775480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4" event={"ID":"acfc4afb-8b3f-4337-b730-6d0f542b0472","Type":"ContainerStarted","Data":"5a53d25cffd9a41332cb5d4f0e1c523a75a41315b9c28eace5e4b3e7f51ede92"}
Sep 16 19:16:22.797832 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:22.797794    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" containerID="7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd" exitCode=0
Sep 16 19:16:22.798285 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:22.797869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" event={"ID":"1d01764a-9d72-48ed-8ab7-d7db4a4405ef","Type":"ContainerDied","Data":"7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd"}
Sep 16 19:16:22.799569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:22.799551    2580 generic.go:356] "Generic (PLEG): container finished" podID="acfc4afb-8b3f-4337-b730-6d0f542b0472" containerID="86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52" exitCode=0
Sep 16 19:16:22.799676 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:22.799625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4" event={"ID":"acfc4afb-8b3f-4337-b730-6d0f542b0472","Type":"ContainerDied","Data":"86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52"}
Sep 16 19:16:23.807411 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:23.807377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" event={"ID":"1d01764a-9d72-48ed-8ab7-d7db4a4405ef","Type":"ContainerStarted","Data":"057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7"}
Sep 16 19:16:23.809519 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:23.809496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4" event={"ID":"acfc4afb-8b3f-4337-b730-6d0f542b0472","Type":"ContainerStarted","Data":"3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c"}
Sep 16 19:16:23.809734 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:23.809713    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:16:23.838795 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:23.838742    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" podStartSLOduration=3.335215626 podStartE2EDuration="5.838725561s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.503509935s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:23.837438332 +0000 UTC m=+1964.099115345" watchObservedRunningTime="2026-09-16 19:16:23.838725561 +0000 UTC m=+1964.100402570"
Sep 16 19:16:23.869443 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:23.869387    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4" podStartSLOduration=3.329750232 podStartE2EDuration="5.869369371s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.539619139s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:23.868886011 +0000 UTC m=+1964.130563027" watchObservedRunningTime="2026-09-16 19:16:23.869369371 +0000 UTC m=+1964.131046377"
Sep 16 19:16:23.936312 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:23.936279    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:28.936246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:28.936200    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:28.948649 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:28.948621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:29.851348 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:29.851315    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:16:54.826323 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:16:54.826285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:17:01.283585 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:01.283550    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"]
Sep 16 19:17:06.744819 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:06.744784    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4_acfc4afb-8b3f-4337-b730-6d0f542b0472/main/0.log"
Sep 16 19:17:16.303416 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.303352    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk" podUID="1f95169d-db02-47bd-8ab5-b37290b51441" containerName="main" containerID="cri-o://f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971" gracePeriod=45
Sep 16 19:17:16.539932 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.539905    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:17:16.589609 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.589549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f95169d-db02-47bd-8ab5-b37290b51441-tls-certs\") pod \"1f95169d-db02-47bd-8ab5-b37290b51441\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") "
Sep 16 19:17:16.589609 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.589600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-dshm\") pod \"1f95169d-db02-47bd-8ab5-b37290b51441\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") "
Sep 16 19:17:16.589774 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.589625    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv4vb\" (UniqueName: \"kubernetes.io/projected/1f95169d-db02-47bd-8ab5-b37290b51441-kube-api-access-fv4vb\") pod \"1f95169d-db02-47bd-8ab5-b37290b51441\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") "
Sep 16 19:17:16.589774 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.589654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-tmp-dir\") pod \"1f95169d-db02-47bd-8ab5-b37290b51441\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") "
Sep 16 19:17:16.589774 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.589679    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-home\") pod \"1f95169d-db02-47bd-8ab5-b37290b51441\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") "
Sep 16 19:17:16.589774 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.589737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-model-cache\") pod \"1f95169d-db02-47bd-8ab5-b37290b51441\" (UID: \"1f95169d-db02-47bd-8ab5-b37290b51441\") "
Sep 16 19:17:16.590048 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.589997    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f95169d-db02-47bd-8ab5-b37290b51441" (UID: "1f95169d-db02-47bd-8ab5-b37290b51441"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:16.590048 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.590002    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-home" (OuterVolumeSpecName: "home") pod "1f95169d-db02-47bd-8ab5-b37290b51441" (UID: "1f95169d-db02-47bd-8ab5-b37290b51441"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:16.590225 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.590057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f95169d-db02-47bd-8ab5-b37290b51441" (UID: "1f95169d-db02-47bd-8ab5-b37290b51441"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:16.592011 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.591989    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f95169d-db02-47bd-8ab5-b37290b51441-kube-api-access-fv4vb" (OuterVolumeSpecName: "kube-api-access-fv4vb") pod "1f95169d-db02-47bd-8ab5-b37290b51441" (UID: "1f95169d-db02-47bd-8ab5-b37290b51441"). InnerVolumeSpecName "kube-api-access-fv4vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:16.592113 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.592020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-dshm" (OuterVolumeSpecName: "dshm") pod "1f95169d-db02-47bd-8ab5-b37290b51441" (UID: "1f95169d-db02-47bd-8ab5-b37290b51441"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:16.592113 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.592027    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f95169d-db02-47bd-8ab5-b37290b51441-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f95169d-db02-47bd-8ab5-b37290b51441" (UID: "1f95169d-db02-47bd-8ab5-b37290b51441"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:16.690752 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.690725    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:16.690752 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.690754    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f95169d-db02-47bd-8ab5-b37290b51441-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:16.690921 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.690764    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:16.690921 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.690773    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv4vb\" (UniqueName: \"kubernetes.io/projected/1f95169d-db02-47bd-8ab5-b37290b51441-kube-api-access-fv4vb\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:16.690921 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.690783    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:16.690921 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:16.690791    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f95169d-db02-47bd-8ab5-b37290b51441-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:17.082528 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:17.082496    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f95169d-db02-47bd-8ab5-b37290b51441" containerID="f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971" exitCode=0
Sep 16 19:17:17.082727 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:17.082551    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"
Sep 16 19:17:17.082727 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:17.082585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk" event={"ID":"1f95169d-db02-47bd-8ab5-b37290b51441","Type":"ContainerDied","Data":"f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971"}
Sep 16 19:17:17.082727 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:17.082643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk" event={"ID":"1f95169d-db02-47bd-8ab5-b37290b51441","Type":"ContainerDied","Data":"b8399b19a960ae5d68f80f36d7f2bea8ad81db65864171d2fd57923efb261a7e"}
Sep 16 19:17:17.082727 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:17.082667    2580 scope.go:117] "RemoveContainer" containerID="f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971"
Sep 16 19:17:17.091733 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:17.091713    2580 scope.go:117] "RemoveContainer" containerID="f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971"
Sep 16 19:17:17.092029 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:17:17.092011    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971\": container with ID starting with f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971 not found: ID does not exist" containerID="f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971"
Sep 16 19:17:17.092096 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:17.092038    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971"} err="failed to get container status \"f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971\": rpc error: code = NotFound desc = could not find container \"f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971\": container with ID starting with f6751269babfa73adfdabe0f2c19989cb6eaedcf4104a1982d7d9430d57a9971 not found: ID does not exist"
Sep 16 19:17:17.112028 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:17.111503    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"]
Sep 16 19:17:17.116371 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:17.116339    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-wwnzk"]
Sep 16 19:17:18.275963 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:18.275930    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f95169d-db02-47bd-8ab5-b37290b51441" path="/var/lib/kubelet/pods/1f95169d-db02-47bd-8ab5-b37290b51441/volumes"
Sep 16 19:17:47.183964 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.183929    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"]
Sep 16 19:17:47.194156 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:17:47.194121    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:17:47.194156 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:47.194156 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:47.194156 ip-10-0-139-95 kubenswrapper[2580]:  > containerID="057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7" cmd=["/bin/sleep","15"]
Sep 16 19:17:47.194419 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.194174    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" message="" err=<
Sep 16 19:17:47.194419 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:47.194419 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:47.194419 ip-10-0-139-95 kubenswrapper[2580]:  >
Sep 16 19:17:47.194419 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:17:47.194188    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:17:47.194419 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:47.194419 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:47.194419 ip-10-0-139-95 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" podUID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" containerName="main" containerID="cri-o://057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7"
Sep 16 19:17:47.194419 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.194228    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" podUID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" containerName="main" containerID="cri-o://057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7" gracePeriod=60
Sep 16 19:17:47.204815 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.204791    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"]
Sep 16 19:17:47.205020 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.204999    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4" podUID="acfc4afb-8b3f-4337-b730-6d0f542b0472" containerName="main" containerID="cri-o://3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c" gracePeriod=30
Sep 16 19:17:47.434088 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.434022    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:17:47.575134 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.575102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tls-certs\") pod \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") "
Sep 16 19:17:47.575447 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.575157    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-home\") pod \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") "
Sep 16 19:17:47.575447 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.575196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qp6h\" (UniqueName: \"kubernetes.io/projected/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kube-api-access-5qp6h\") pod \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") "
Sep 16 19:17:47.575447 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.575397    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-home" (OuterVolumeSpecName: "home") pod "1d01764a-9d72-48ed-8ab7-d7db4a4405ef" (UID: "1d01764a-9d72-48ed-8ab7-d7db4a4405ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:47.575640 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.575531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-dshm\") pod \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") "
Sep 16 19:17:47.575640 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.575578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-cabundle-cert\") pod \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") "
Sep 16 19:17:47.575640 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.575616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-model-cache\") pod \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") "
Sep 16 19:17:47.575802 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.575686    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kserve-provision-location\") pod \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") "
Sep 16 19:17:47.575802 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.575720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tmp-dir\") pod \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\" (UID: \"1d01764a-9d72-48ed-8ab7-d7db4a4405ef\") "
Sep 16 19:17:47.577947 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.576025    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:47.577947 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.576142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1d01764a-9d72-48ed-8ab7-d7db4a4405ef" (UID: "1d01764a-9d72-48ed-8ab7-d7db4a4405ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:47.577947 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.576201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d01764a-9d72-48ed-8ab7-d7db4a4405ef" (UID: "1d01764a-9d72-48ed-8ab7-d7db4a4405ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:47.577947 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.576358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d01764a-9d72-48ed-8ab7-d7db4a4405ef" (UID: "1d01764a-9d72-48ed-8ab7-d7db4a4405ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:47.578260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.578181    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kube-api-access-5qp6h" (OuterVolumeSpecName: "kube-api-access-5qp6h") pod "1d01764a-9d72-48ed-8ab7-d7db4a4405ef" (UID: "1d01764a-9d72-48ed-8ab7-d7db4a4405ef"). InnerVolumeSpecName "kube-api-access-5qp6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:47.578319 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.578262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d01764a-9d72-48ed-8ab7-d7db4a4405ef" (UID: "1d01764a-9d72-48ed-8ab7-d7db4a4405ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:47.591059 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.591030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-dshm" (OuterVolumeSpecName: "dshm") pod "1d01764a-9d72-48ed-8ab7-d7db4a4405ef" (UID: "1d01764a-9d72-48ed-8ab7-d7db4a4405ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:47.610232 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.610203    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d01764a-9d72-48ed-8ab7-d7db4a4405ef" (UID: "1d01764a-9d72-48ed-8ab7-d7db4a4405ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:47.677149 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.677114    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qp6h\" (UniqueName: \"kubernetes.io/projected/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kube-api-access-5qp6h\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:47.677291 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.677155    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:47.677291 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.677173    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:47.677291 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.677186    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:47.677291 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.677199    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:47.677291 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.677212    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:47.677291 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.677226    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d01764a-9d72-48ed-8ab7-d7db4a4405ef-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:47.841278 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.841254    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:17:47.980550 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.980519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-kserve-provision-location\") pod \"acfc4afb-8b3f-4337-b730-6d0f542b0472\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") "
Sep 16 19:17:47.980738 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.980601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acfc4afb-8b3f-4337-b730-6d0f542b0472-cabundle-cert\") pod \"acfc4afb-8b3f-4337-b730-6d0f542b0472\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") "
Sep 16 19:17:47.980738 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.980638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acfc4afb-8b3f-4337-b730-6d0f542b0472-tls-certs\") pod \"acfc4afb-8b3f-4337-b730-6d0f542b0472\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") "
Sep 16 19:17:47.980738 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.980672    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-tmp-dir\") pod \"acfc4afb-8b3f-4337-b730-6d0f542b0472\" (UID: \"acfc4afb-8b3f-4337-b730-6d0f542b0472\") "
Sep 16 19:17:47.981077 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.981018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfc4afb-8b3f-4337-b730-6d0f542b0472-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "acfc4afb-8b3f-4337-b730-6d0f542b0472" (UID: "acfc4afb-8b3f-4337-b730-6d0f542b0472"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:47.981157 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.981118    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acfc4afb-8b3f-4337-b730-6d0f542b0472" (UID: "acfc4afb-8b3f-4337-b730-6d0f542b0472"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:47.982823 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:47.982799    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfc4afb-8b3f-4337-b730-6d0f542b0472-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acfc4afb-8b3f-4337-b730-6d0f542b0472" (UID: "acfc4afb-8b3f-4337-b730-6d0f542b0472"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:48.003686 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.003660    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "acfc4afb-8b3f-4337-b730-6d0f542b0472" (UID: "acfc4afb-8b3f-4337-b730-6d0f542b0472"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.082185 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.082162    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.082185 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.082186    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acfc4afb-8b3f-4337-b730-6d0f542b0472-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.082327 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.082197    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acfc4afb-8b3f-4337-b730-6d0f542b0472-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.082327 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.082206    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acfc4afb-8b3f-4337-b730-6d0f542b0472-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.235791 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.235712    2580 generic.go:356] "Generic (PLEG): container finished" podID="acfc4afb-8b3f-4337-b730-6d0f542b0472" containerID="3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c" exitCode=0
Sep 16 19:17:48.236224 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.235818    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"
Sep 16 19:17:48.236224 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.235829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4" event={"ID":"acfc4afb-8b3f-4337-b730-6d0f542b0472","Type":"ContainerDied","Data":"3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c"}
Sep 16 19:17:48.236224 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.235869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4" event={"ID":"acfc4afb-8b3f-4337-b730-6d0f542b0472","Type":"ContainerDied","Data":"5a53d25cffd9a41332cb5d4f0e1c523a75a41315b9c28eace5e4b3e7f51ede92"}
Sep 16 19:17:48.236224 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.235913    2580 scope.go:117] "RemoveContainer" containerID="3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c"
Sep 16 19:17:48.237630 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.237611    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" containerID="057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7" exitCode=0
Sep 16 19:17:48.237726 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.237638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" event={"ID":"1d01764a-9d72-48ed-8ab7-d7db4a4405ef","Type":"ContainerDied","Data":"057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7"}
Sep 16 19:17:48.237726 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.237664    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"
Sep 16 19:17:48.237842 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.237666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx" event={"ID":"1d01764a-9d72-48ed-8ab7-d7db4a4405ef","Type":"ContainerDied","Data":"6452056f902c7531c3b505218385f332ebafb04026633611ac7d324927f6030c"}
Sep 16 19:17:48.244746 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.244730    2580 scope.go:117] "RemoveContainer" containerID="86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52"
Sep 16 19:17:48.258010 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.257979    2580 scope.go:117] "RemoveContainer" containerID="3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c"
Sep 16 19:17:48.258628 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:17:48.258593    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c\": container with ID starting with 3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c not found: ID does not exist" containerID="3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c"
Sep 16 19:17:48.258747 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.258632    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c"} err="failed to get container status \"3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c\": rpc error: code = NotFound desc = could not find container \"3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c\": container with ID starting with 3a9b2cf7d1403d1a5dca5fae25fc873d071d933ff8bf280922c13ff3b4906f8c not found: ID does not exist"
Sep 16 19:17:48.258747 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.258657    2580 scope.go:117] "RemoveContainer" containerID="86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52"
Sep 16 19:17:48.259089 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:17:48.259051    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52\": container with ID starting with 86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52 not found: ID does not exist" containerID="86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52"
Sep 16 19:17:48.259143 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.259106    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52"} err="failed to get container status \"86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52\": rpc error: code = NotFound desc = could not find container \"86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52\": container with ID starting with 86716d7e2de7c3ec5709c7a8623b78b2ce2548bcee419ad3b6a7d27df3979c52 not found: ID does not exist"
Sep 16 19:17:48.259189 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.259146    2580 scope.go:117] "RemoveContainer" containerID="057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7"
Sep 16 19:17:48.260677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.260657    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"]
Sep 16 19:17:48.264687 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.264665    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-b88cd859d-k2qv4"]
Sep 16 19:17:48.269517 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.269495    2580 scope.go:117] "RemoveContainer" containerID="7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd"
Sep 16 19:17:48.276262 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.276237    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acfc4afb-8b3f-4337-b730-6d0f542b0472" path="/var/lib/kubelet/pods/acfc4afb-8b3f-4337-b730-6d0f542b0472/volumes"
Sep 16 19:17:48.276533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.276518    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"]
Sep 16 19:17:48.278198 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.277886    2580 scope.go:117] "RemoveContainer" containerID="057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7"
Sep 16 19:17:48.278516 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:17:48.278489    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7\": container with ID starting with 057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7 not found: ID does not exist" containerID="057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7"
Sep 16 19:17:48.278620 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.278527    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7"} err="failed to get container status \"057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7\": rpc error: code = NotFound desc = could not find container \"057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7\": container with ID starting with 057abd8153f8b726fc034f9d217b5be7079d7025579816a1e8a053a9bb13ecf7 not found: ID does not exist"
Sep 16 19:17:48.278620 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.278550    2580 scope.go:117] "RemoveContainer" containerID="7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd"
Sep 16 19:17:48.278854 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:17:48.278837    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd\": container with ID starting with 7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd not found: ID does not exist" containerID="7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd"
Sep 16 19:17:48.279180 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.278860    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd"} err="failed to get container status \"7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd\": rpc error: code = NotFound desc = could not find container \"7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd\": container with ID starting with 7c8bf1c7a64895a7e726340885069eff13849d60451aef377aa75df854331ffd not found: ID does not exist"
Sep 16 19:17:48.279611 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:48.279592    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-774757bb8d-8fgrx"]
Sep 16 19:17:50.275602 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:17:50.275572    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" path="/var/lib/kubelet/pods/1d01764a-9d72-48ed-8ab7-d7db4a4405ef/volumes"
Sep 16 19:18:44.812809 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.812771    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"]
Sep 16 19:18:44.813214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813158    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acfc4afb-8b3f-4337-b730-6d0f542b0472" containerName="storage-initializer"
Sep 16 19:18:44.813214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813172    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="acfc4afb-8b3f-4337-b730-6d0f542b0472" containerName="storage-initializer"
Sep 16 19:18:44.813214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813186    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f95169d-db02-47bd-8ab5-b37290b51441" containerName="main"
Sep 16 19:18:44.813214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813191    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f95169d-db02-47bd-8ab5-b37290b51441" containerName="main"
Sep 16 19:18:44.813214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813199    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" containerName="main"
Sep 16 19:18:44.813214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813204    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" containerName="main"
Sep 16 19:18:44.813214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813211    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acfc4afb-8b3f-4337-b730-6d0f542b0472" containerName="main"
Sep 16 19:18:44.813214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813216    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="acfc4afb-8b3f-4337-b730-6d0f542b0472" containerName="main"
Sep 16 19:18:44.813463 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813233    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" containerName="storage-initializer"
Sep 16 19:18:44.813463 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813238    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" containerName="storage-initializer"
Sep 16 19:18:44.813463 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813289    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="acfc4afb-8b3f-4337-b730-6d0f542b0472" containerName="main"
Sep 16 19:18:44.813463 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813300    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d01764a-9d72-48ed-8ab7-d7db4a4405ef" containerName="main"
Sep 16 19:18:44.813463 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.813306    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f95169d-db02-47bd-8ab5-b37290b51441" containerName="main"
Sep 16 19:18:44.816091 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.816073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.819150 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.819132    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 19:18:44.819266 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.819150    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 19:18:44.819708 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.819689    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 16 19:18:44.819831 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.819772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-9mt2q\""
Sep 16 19:18:44.826862 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.826839    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"]
Sep 16 19:18:44.867342 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.867314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-home\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.867441 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.867349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.867441 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.867368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-dshm\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.867441 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.867436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-model-cache\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.867549 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.867458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005dace5-f43e-468f-9240-00ed01738c70-tls-certs\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.867549 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.867502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m79vc\" (UniqueName: \"kubernetes.io/projected/005dace5-f43e-468f-9240-00ed01738c70-kube-api-access-m79vc\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.967850 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.967826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79vc\" (UniqueName: \"kubernetes.io/projected/005dace5-f43e-468f-9240-00ed01738c70-kube-api-access-m79vc\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.967989 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.967864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-home\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.967989 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.967882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.967989 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.967917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-dshm\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.967989 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.967957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-model-cache\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.967989 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.967985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005dace5-f43e-468f-9240-00ed01738c70-tls-certs\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.968336 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.968313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-home\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.968439 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.968328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.968439 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.968386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-model-cache\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.970468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.970449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-dshm\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.970634 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.970618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005dace5-f43e-468f-9240-00ed01738c70-tls-certs\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:44.977289 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:44.977267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79vc\" (UniqueName: \"kubernetes.io/projected/005dace5-f43e-468f-9240-00ed01738c70-kube-api-access-m79vc\") pod \"leave-v1-kserve-65684d4f5-j8bsj\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:45.126362 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:45.126296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:45.403640 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:45.403609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"]
Sep 16 19:18:45.405938 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:18:45.405910    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005dace5_f43e_468f_9240_00ed01738c70.slice/crio-435a6e389201c41a930f9ea93480ba080ef5b47047b3f18f35dd8603d7ad0947 WatchSource:0}: Error finding container 435a6e389201c41a930f9ea93480ba080ef5b47047b3f18f35dd8603d7ad0947: Status 404 returned error can't find the container with id 435a6e389201c41a930f9ea93480ba080ef5b47047b3f18f35dd8603d7ad0947
Sep 16 19:18:45.501698 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:45.501665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj" event={"ID":"005dace5-f43e-468f-9240-00ed01738c70","Type":"ContainerStarted","Data":"8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189"}
Sep 16 19:18:45.501849 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:45.501724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj" event={"ID":"005dace5-f43e-468f-9240-00ed01738c70","Type":"ContainerStarted","Data":"435a6e389201c41a930f9ea93480ba080ef5b47047b3f18f35dd8603d7ad0947"}
Sep 16 19:18:45.523141 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:45.523094    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj" podStartSLOduration=1.523078366 podStartE2EDuration="1.523078366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:45.522035139 +0000 UTC m=+2105.783712151" watchObservedRunningTime="2026-09-16 19:18:45.523078366 +0000 UTC m=+2105.784755377"
Sep 16 19:18:46.126731 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:46.126693    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:55.126461 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:55.126423    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:55.140042 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:55.140005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:18:55.560850 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:18:55.560814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:20:05.800190 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.800153    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"]
Sep 16 19:20:05.803720 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.803699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:05.811189 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.808234    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:20:05.812338 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.812312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:20:05.813885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.812638    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:20:05.813885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.812810    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:20:05.813885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.812980    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:20:05.813885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.812643    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-c4ct4\""
Sep 16 19:20:05.837573 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.837547    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"]
Sep 16 19:20:05.921609 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.921579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:05.921750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.921611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea52e50f-80e0-4beb-ac49-eace29207018-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:05.921750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.921638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-tmp-dir\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:05.921750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.921689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-model-cache\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:05.921855 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.921767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqv79\" (UniqueName: \"kubernetes.io/projected/ea52e50f-80e0-4beb-ac49-eace29207018-kube-api-access-lqv79\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:05.921855 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.921802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-dshm\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:05.921855 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.921828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-home\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:05.922025 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:05.921909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea52e50f-80e0-4beb-ac49-eace29207018-tls-certs\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.022591 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.022567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-tmp-dir\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.022741 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.022602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-model-cache\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.022741 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.022621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqv79\" (UniqueName: \"kubernetes.io/projected/ea52e50f-80e0-4beb-ac49-eace29207018-kube-api-access-lqv79\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.022741 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.022644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-dshm\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.022741 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.022674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-home\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.022741 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.022728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea52e50f-80e0-4beb-ac49-eace29207018-tls-certs\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.023041 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.022771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.023041 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.022939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea52e50f-80e0-4beb-ac49-eace29207018-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.023041 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.023020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-tmp-dir\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.023189 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.023049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-model-cache\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.023189 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.023158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.023270 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.023196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-home\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.023570 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.023552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea52e50f-80e0-4beb-ac49-eace29207018-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.025203 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.025172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-dshm\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.025463 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.025442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea52e50f-80e0-4beb-ac49-eace29207018-tls-certs\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.032687 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.032662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqv79\" (UniqueName: \"kubernetes.io/projected/ea52e50f-80e0-4beb-ac49-eace29207018-kube-api-access-lqv79\") pod \"tokenizer-template-args-test-kserve-656f6fb7b8-wlm55\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.116502 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.116423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:06.174261 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.174040    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"]
Sep 16 19:20:06.178954 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.178936    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.201413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.201385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"]
Sep 16 19:20:06.224650 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.224620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.224775 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.224658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155749d1-fb68-4fb3-be12-28beb31c639b-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.224775 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.224688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.224775 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.224727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155749d1-fb68-4fb3-be12-28beb31c639b-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.325380 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.325349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.325534 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.325401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155749d1-fb68-4fb3-be12-28beb31c639b-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.325534 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.325502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.325649 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.325545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155749d1-fb68-4fb3-be12-28beb31c639b-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.325777 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.325750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.325858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.325832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.326075 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.326048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155749d1-fb68-4fb3-be12-28beb31c639b-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.328161 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.328138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155749d1-fb68-4fb3-be12-28beb31c639b-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.406882 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.406812    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"]
Sep 16 19:20:06.409695 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:20:06.409666    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea52e50f_80e0_4beb_ac49_eace29207018.slice/crio-5258380adeeb6d5eaffab7e447ab6d830ade0e5f71015832aa3f02eab6671de9 WatchSource:0}: Error finding container 5258380adeeb6d5eaffab7e447ab6d830ade0e5f71015832aa3f02eab6671de9: Status 404 returned error can't find the container with id 5258380adeeb6d5eaffab7e447ab6d830ade0e5f71015832aa3f02eab6671de9
Sep 16 19:20:06.494732 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.494705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:06.783997 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.783953    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"]
Sep 16 19:20:06.784372 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:20:06.784341    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155749d1_fb68_4fb3_be12_28beb31c639b.slice/crio-9a4ef509ed5a536312aa4fcccb1a575698e06d7597c1388e3ed1e6174adb2627 WatchSource:0}: Error finding container 9a4ef509ed5a536312aa4fcccb1a575698e06d7597c1388e3ed1e6174adb2627: Status 404 returned error can't find the container with id 9a4ef509ed5a536312aa4fcccb1a575698e06d7597c1388e3ed1e6174adb2627
Sep 16 19:20:06.886170 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.885792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j" event={"ID":"155749d1-fb68-4fb3-be12-28beb31c639b","Type":"ContainerStarted","Data":"35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec"}
Sep 16 19:20:06.886170 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.885865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j" event={"ID":"155749d1-fb68-4fb3-be12-28beb31c639b","Type":"ContainerStarted","Data":"9a4ef509ed5a536312aa4fcccb1a575698e06d7597c1388e3ed1e6174adb2627"}
Sep 16 19:20:06.887959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.887934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" event={"ID":"ea52e50f-80e0-4beb-ac49-eace29207018","Type":"ContainerStarted","Data":"54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98"}
Sep 16 19:20:06.888079 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:06.887985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" event={"ID":"ea52e50f-80e0-4beb-ac49-eace29207018","Type":"ContainerStarted","Data":"5258380adeeb6d5eaffab7e447ab6d830ade0e5f71015832aa3f02eab6671de9"}
Sep 16 19:20:09.906437 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:09.906400    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea52e50f-80e0-4beb-ac49-eace29207018" containerID="54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98" exitCode=0
Sep 16 19:20:09.906883 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:09.906474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" event={"ID":"ea52e50f-80e0-4beb-ac49-eace29207018","Type":"ContainerDied","Data":"54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98"}
Sep 16 19:20:09.908283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:09.908265    2580 generic.go:356] "Generic (PLEG): container finished" podID="155749d1-fb68-4fb3-be12-28beb31c639b" containerID="35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec" exitCode=0
Sep 16 19:20:09.908393 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:09.908290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j" event={"ID":"155749d1-fb68-4fb3-be12-28beb31c639b","Type":"ContainerDied","Data":"35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec"}
Sep 16 19:20:10.914711 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:10.914672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j" event={"ID":"155749d1-fb68-4fb3-be12-28beb31c639b","Type":"ContainerStarted","Data":"ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798"}
Sep 16 19:20:10.915214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:10.914920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:10.916597 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:10.916573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" event={"ID":"ea52e50f-80e0-4beb-ac49-eace29207018","Type":"ContainerStarted","Data":"b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05"}
Sep 16 19:20:11.001370 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:11.001319    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j" podStartSLOduration=2.571603454 podStartE2EDuration="5.001303763s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.429700309s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:10.999101263 +0000 UTC m=+2191.260778273" watchObservedRunningTime="2026-09-16 19:20:11.001303763 +0000 UTC m=+2191.262980775"
Sep 16 19:20:11.050626 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:11.050581    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" podStartSLOduration=3.590024756 podStartE2EDuration="6.050566728s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.460541972s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:11.046851502 +0000 UTC m=+2191.308528514" watchObservedRunningTime="2026-09-16 19:20:11.050566728 +0000 UTC m=+2191.312243742"
Sep 16 19:20:11.117016 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:11.116983    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:16.116691 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:16.116658    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:16.129258 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:16.129232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:16.961242 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:16.961215    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:41.931859 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:41.931828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:43.729851 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:43.729817    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j_155749d1-fb68-4fb3-be12-28beb31c639b/main/0.log"
Sep 16 19:20:45.870466 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:45.870434    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"]
Sep 16 19:20:45.870871 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:45.870633    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j" podUID="155749d1-fb68-4fb3-be12-28beb31c639b" containerName="main" containerID="cri-o://ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798" gracePeriod=30
Sep 16 19:20:45.892098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:45.892064    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"]
Sep 16 19:20:45.902812 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:20:45.902772    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:20:45.902812 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:20:45.902812 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:20:45.902812 ip-10-0-139-95 kubenswrapper[2580]:  > containerID="b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05" cmd=["/bin/sleep","15"]
Sep 16 19:20:45.903120 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:45.902832    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" message="" err=<
Sep 16 19:20:45.903120 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:20:45.903120 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:20:45.903120 ip-10-0-139-95 kubenswrapper[2580]:  >
Sep 16 19:20:45.903120 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:20:45.902849    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:20:45.903120 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:20:45.903120 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:20:45.903120 ip-10-0-139-95 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" podUID="ea52e50f-80e0-4beb-ac49-eace29207018" containerName="main" containerID="cri-o://b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05"
Sep 16 19:20:45.903120 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:45.902924    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" podUID="ea52e50f-80e0-4beb-ac49-eace29207018" containerName="main" containerID="cri-o://b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05" gracePeriod=60
Sep 16 19:20:45.950352 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:45.950313    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" podUID="ea52e50f-80e0-4beb-ac49-eace29207018" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 16 19:20:46.260126 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.260099    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:46.295823 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.295787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea52e50f-80e0-4beb-ac49-eace29207018-cabundle-cert\") pod \"ea52e50f-80e0-4beb-ac49-eace29207018\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") "
Sep 16 19:20:46.296033 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.295827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-home\") pod \"ea52e50f-80e0-4beb-ac49-eace29207018\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") "
Sep 16 19:20:46.296033 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.295877    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqv79\" (UniqueName: \"kubernetes.io/projected/ea52e50f-80e0-4beb-ac49-eace29207018-kube-api-access-lqv79\") pod \"ea52e50f-80e0-4beb-ac49-eace29207018\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") "
Sep 16 19:20:46.296033 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.295935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea52e50f-80e0-4beb-ac49-eace29207018-tls-certs\") pod \"ea52e50f-80e0-4beb-ac49-eace29207018\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") "
Sep 16 19:20:46.296236 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296098    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-home" (OuterVolumeSpecName: "home") pod "ea52e50f-80e0-4beb-ac49-eace29207018" (UID: "ea52e50f-80e0-4beb-ac49-eace29207018"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.296236 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296148    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-dshm\") pod \"ea52e50f-80e0-4beb-ac49-eace29207018\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") "
Sep 16 19:20:46.296236 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296160    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea52e50f-80e0-4beb-ac49-eace29207018-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea52e50f-80e0-4beb-ac49-eace29207018" (UID: "ea52e50f-80e0-4beb-ac49-eace29207018"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:46.296236 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296192    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-tmp-dir\") pod \"ea52e50f-80e0-4beb-ac49-eace29207018\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") "
Sep 16 19:20:46.296470 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-kserve-provision-location\") pod \"ea52e50f-80e0-4beb-ac49-eace29207018\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") "
Sep 16 19:20:46.296470 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-model-cache\") pod \"ea52e50f-80e0-4beb-ac49-eace29207018\" (UID: \"ea52e50f-80e0-4beb-ac49-eace29207018\") "
Sep 16 19:20:46.296579 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296466    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea52e50f-80e0-4beb-ac49-eace29207018" (UID: "ea52e50f-80e0-4beb-ac49-eace29207018"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.296726 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296706    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.296825 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296730    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea52e50f-80e0-4beb-ac49-eace29207018-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.296825 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296746    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.296825 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.296776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea52e50f-80e0-4beb-ac49-eace29207018" (UID: "ea52e50f-80e0-4beb-ac49-eace29207018"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.298597 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.298564    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-dshm" (OuterVolumeSpecName: "dshm") pod "ea52e50f-80e0-4beb-ac49-eace29207018" (UID: "ea52e50f-80e0-4beb-ac49-eace29207018"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.298704 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.298662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea52e50f-80e0-4beb-ac49-eace29207018-kube-api-access-lqv79" (OuterVolumeSpecName: "kube-api-access-lqv79") pod "ea52e50f-80e0-4beb-ac49-eace29207018" (UID: "ea52e50f-80e0-4beb-ac49-eace29207018"). InnerVolumeSpecName "kube-api-access-lqv79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:46.299185 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.299162    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea52e50f-80e0-4beb-ac49-eace29207018-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea52e50f-80e0-4beb-ac49-eace29207018" (UID: "ea52e50f-80e0-4beb-ac49-eace29207018"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:46.321070 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.321043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea52e50f-80e0-4beb-ac49-eace29207018" (UID: "ea52e50f-80e0-4beb-ac49-eace29207018"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.398179 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.398155    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.398179 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.398180    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.398335 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.398191    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqv79\" (UniqueName: \"kubernetes.io/projected/ea52e50f-80e0-4beb-ac49-eace29207018-kube-api-access-lqv79\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.398335 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.398201    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea52e50f-80e0-4beb-ac49-eace29207018-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.398335 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.398210    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea52e50f-80e0-4beb-ac49-eace29207018-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.503309 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.503282    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:46.600044 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.600016    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-kserve-provision-location\") pod \"155749d1-fb68-4fb3-be12-28beb31c639b\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") "
Sep 16 19:20:46.600191 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.600057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155749d1-fb68-4fb3-be12-28beb31c639b-cabundle-cert\") pod \"155749d1-fb68-4fb3-be12-28beb31c639b\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") "
Sep 16 19:20:46.600191 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.600101    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-tmp-dir\") pod \"155749d1-fb68-4fb3-be12-28beb31c639b\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") "
Sep 16 19:20:46.600191 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.600122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155749d1-fb68-4fb3-be12-28beb31c639b-tls-certs\") pod \"155749d1-fb68-4fb3-be12-28beb31c639b\" (UID: \"155749d1-fb68-4fb3-be12-28beb31c639b\") "
Sep 16 19:20:46.600435 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.600413    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155749d1-fb68-4fb3-be12-28beb31c639b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "155749d1-fb68-4fb3-be12-28beb31c639b" (UID: "155749d1-fb68-4fb3-be12-28beb31c639b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:46.600502 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.600465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "155749d1-fb68-4fb3-be12-28beb31c639b" (UID: "155749d1-fb68-4fb3-be12-28beb31c639b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.602491 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.602470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155749d1-fb68-4fb3-be12-28beb31c639b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "155749d1-fb68-4fb3-be12-28beb31c639b" (UID: "155749d1-fb68-4fb3-be12-28beb31c639b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:46.623046 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.623019    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "155749d1-fb68-4fb3-be12-28beb31c639b" (UID: "155749d1-fb68-4fb3-be12-28beb31c639b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:46.701284 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.701225    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.701284 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.701247    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155749d1-fb68-4fb3-be12-28beb31c639b-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.701284 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.701258    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155749d1-fb68-4fb3-be12-28beb31c639b-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.701284 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:46.701266    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155749d1-fb68-4fb3-be12-28beb31c639b-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:20:47.108024 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.107991    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea52e50f-80e0-4beb-ac49-eace29207018" containerID="b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05" exitCode=0
Sep 16 19:20:47.108454 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.108048    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"
Sep 16 19:20:47.108454 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.108071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" event={"ID":"ea52e50f-80e0-4beb-ac49-eace29207018","Type":"ContainerDied","Data":"b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05"}
Sep 16 19:20:47.108454 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.108115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55" event={"ID":"ea52e50f-80e0-4beb-ac49-eace29207018","Type":"ContainerDied","Data":"5258380adeeb6d5eaffab7e447ab6d830ade0e5f71015832aa3f02eab6671de9"}
Sep 16 19:20:47.108454 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.108138    2580 scope.go:117] "RemoveContainer" containerID="b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05"
Sep 16 19:20:47.109690 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.109670    2580 generic.go:356] "Generic (PLEG): container finished" podID="155749d1-fb68-4fb3-be12-28beb31c639b" containerID="ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798" exitCode=0
Sep 16 19:20:47.109782 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.109702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j" event={"ID":"155749d1-fb68-4fb3-be12-28beb31c639b","Type":"ContainerDied","Data":"ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798"}
Sep 16 19:20:47.109782 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.109722    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"
Sep 16 19:20:47.109914 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.109729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j" event={"ID":"155749d1-fb68-4fb3-be12-28beb31c639b","Type":"ContainerDied","Data":"9a4ef509ed5a536312aa4fcccb1a575698e06d7597c1388e3ed1e6174adb2627"}
Sep 16 19:20:47.118124 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.118109    2580 scope.go:117] "RemoveContainer" containerID="54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98"
Sep 16 19:20:47.125836 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.125820    2580 scope.go:117] "RemoveContainer" containerID="b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05"
Sep 16 19:20:47.126659 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:20:47.126639    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05\": container with ID starting with b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05 not found: ID does not exist" containerID="b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05"
Sep 16 19:20:47.126710 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.126667    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05"} err="failed to get container status \"b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05\": rpc error: code = NotFound desc = could not find container \"b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05\": container with ID starting with b047dc9d2897761c7f3aee5bfa637e7caaf3b63cd966185b297d0b2292b44a05 not found: ID does not exist"
Sep 16 19:20:47.126710 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.126682    2580 scope.go:117] "RemoveContainer" containerID="54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98"
Sep 16 19:20:47.126958 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:20:47.126928    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98\": container with ID starting with 54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98 not found: ID does not exist" containerID="54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98"
Sep 16 19:20:47.127048 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.126965    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98"} err="failed to get container status \"54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98\": rpc error: code = NotFound desc = could not find container \"54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98\": container with ID starting with 54e190aee01d83537bf20baa8823fba0f7d0edd2b9b5da9c9e68330224ceec98 not found: ID does not exist"
Sep 16 19:20:47.127048 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.126988    2580 scope.go:117] "RemoveContainer" containerID="ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798"
Sep 16 19:20:47.135347 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.135322    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"]
Sep 16 19:20:47.137873 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.137845    2580 scope.go:117] "RemoveContainer" containerID="35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec"
Sep 16 19:20:47.139979 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.139956    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b79f5c978-wcq6j"]
Sep 16 19:20:47.146197 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.146181    2580 scope.go:117] "RemoveContainer" containerID="ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798"
Sep 16 19:20:47.146470 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:20:47.146453    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798\": container with ID starting with ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798 not found: ID does not exist" containerID="ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798"
Sep 16 19:20:47.146534 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.146476    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798"} err="failed to get container status \"ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798\": rpc error: code = NotFound desc = could not find container \"ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798\": container with ID starting with ee11fd5fdfc180c08b4143f382ba8a07a44c49cc630a3645668cb1db771c5798 not found: ID does not exist"
Sep 16 19:20:47.146534 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.146493    2580 scope.go:117] "RemoveContainer" containerID="35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec"
Sep 16 19:20:47.146718 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:20:47.146703    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec\": container with ID starting with 35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec not found: ID does not exist" containerID="35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec"
Sep 16 19:20:47.146758 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.146723    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec"} err="failed to get container status \"35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec\": rpc error: code = NotFound desc = could not find container \"35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec\": container with ID starting with 35e584ed488c29bbd825c280b212d85781824a2f3d90179f8081d00520cb39ec not found: ID does not exist"
Sep 16 19:20:47.153849 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.153829    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"]
Sep 16 19:20:47.158276 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:47.158252    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-656f6fb7b8-wlm55"]
Sep 16 19:20:48.276036 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:48.276000    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="155749d1-fb68-4fb3-be12-28beb31c639b" path="/var/lib/kubelet/pods/155749d1-fb68-4fb3-be12-28beb31c639b/volumes"
Sep 16 19:20:48.276481 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:48.276466    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea52e50f-80e0-4beb-ac49-eace29207018" path="/var/lib/kubelet/pods/ea52e50f-80e0-4beb-ac49-eace29207018/volumes"
Sep 16 19:20:52.238782 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:20:52.238749    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"]
Sep 16 19:21:07.257934 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.257834    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj" podUID="005dace5-f43e-468f-9240-00ed01738c70" containerName="main" containerID="cri-o://8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189" gracePeriod=45
Sep 16 19:21:07.509261 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.509206    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:21:07.575213 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575177    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005dace5-f43e-468f-9240-00ed01738c70-tls-certs\") pod \"005dace5-f43e-468f-9240-00ed01738c70\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") "
Sep 16 19:21:07.575358 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-home\") pod \"005dace5-f43e-468f-9240-00ed01738c70\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") "
Sep 16 19:21:07.575358 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m79vc\" (UniqueName: \"kubernetes.io/projected/005dace5-f43e-468f-9240-00ed01738c70-kube-api-access-m79vc\") pod \"005dace5-f43e-468f-9240-00ed01738c70\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") "
Sep 16 19:21:07.575358 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-tmp-dir\") pod \"005dace5-f43e-468f-9240-00ed01738c70\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") "
Sep 16 19:21:07.575358 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575350    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-dshm\") pod \"005dace5-f43e-468f-9240-00ed01738c70\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") "
Sep 16 19:21:07.575575 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-model-cache\") pod \"005dace5-f43e-468f-9240-00ed01738c70\" (UID: \"005dace5-f43e-468f-9240-00ed01738c70\") "
Sep 16 19:21:07.575639 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575532    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-home" (OuterVolumeSpecName: "home") pod "005dace5-f43e-468f-9240-00ed01738c70" (UID: "005dace5-f43e-468f-9240-00ed01738c70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:07.575639 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "005dace5-f43e-468f-9240-00ed01738c70" (UID: "005dace5-f43e-468f-9240-00ed01738c70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:07.575742 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575657    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.575821 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.575760    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-model-cache" (OuterVolumeSpecName: "model-cache") pod "005dace5-f43e-468f-9240-00ed01738c70" (UID: "005dace5-f43e-468f-9240-00ed01738c70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:07.577519 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.577494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-dshm" (OuterVolumeSpecName: "dshm") pod "005dace5-f43e-468f-9240-00ed01738c70" (UID: "005dace5-f43e-468f-9240-00ed01738c70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:07.577519 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.577509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005dace5-f43e-468f-9240-00ed01738c70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "005dace5-f43e-468f-9240-00ed01738c70" (UID: "005dace5-f43e-468f-9240-00ed01738c70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:07.577652 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.577602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005dace5-f43e-468f-9240-00ed01738c70-kube-api-access-m79vc" (OuterVolumeSpecName: "kube-api-access-m79vc") pod "005dace5-f43e-468f-9240-00ed01738c70" (UID: "005dace5-f43e-468f-9240-00ed01738c70"). InnerVolumeSpecName "kube-api-access-m79vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:07.677072 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.677045    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m79vc\" (UniqueName: \"kubernetes.io/projected/005dace5-f43e-468f-9240-00ed01738c70-kube-api-access-m79vc\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.677072 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.677068    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.677072 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.677077    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.677253 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.677085    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005dace5-f43e-468f-9240-00ed01738c70-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.677253 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:07.677093    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005dace5-f43e-468f-9240-00ed01738c70-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:21:08.216468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.216427    2580 generic.go:356] "Generic (PLEG): container finished" podID="005dace5-f43e-468f-9240-00ed01738c70" containerID="8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189" exitCode=0
Sep 16 19:21:08.216662 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.216483    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"
Sep 16 19:21:08.216662 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.216519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj" event={"ID":"005dace5-f43e-468f-9240-00ed01738c70","Type":"ContainerDied","Data":"8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189"}
Sep 16 19:21:08.216662 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.216599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj" event={"ID":"005dace5-f43e-468f-9240-00ed01738c70","Type":"ContainerDied","Data":"435a6e389201c41a930f9ea93480ba080ef5b47047b3f18f35dd8603d7ad0947"}
Sep 16 19:21:08.216662 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.216623    2580 scope.go:117] "RemoveContainer" containerID="8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189"
Sep 16 19:21:08.227110 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.227092    2580 scope.go:117] "RemoveContainer" containerID="8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189"
Sep 16 19:21:08.227411 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:21:08.227390    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189\": container with ID starting with 8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189 not found: ID does not exist" containerID="8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189"
Sep 16 19:21:08.227469 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.227419    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189"} err="failed to get container status \"8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189\": rpc error: code = NotFound desc = could not find container \"8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189\": container with ID starting with 8fc8b0024b615bf8fcf639c4b1f5f4e68dccdee2803478e56e5bee5cd4632189 not found: ID does not exist"
Sep 16 19:21:08.241425 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.241400    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"]
Sep 16 19:21:08.245884 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.245863    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-j8bsj"]
Sep 16 19:21:08.275668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:08.275644    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="005dace5-f43e-468f-9240-00ed01738c70" path="/var/lib/kubelet/pods/005dace5-f43e-468f-9240-00ed01738c70/volumes"
Sep 16 19:21:12.804928 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.804872    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"]
Sep 16 19:21:12.805431 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805402    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea52e50f-80e0-4beb-ac49-eace29207018" containerName="main"
Sep 16 19:21:12.805431 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805429    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea52e50f-80e0-4beb-ac49-eace29207018" containerName="main"
Sep 16 19:21:12.805598 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805443    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155749d1-fb68-4fb3-be12-28beb31c639b" containerName="main"
Sep 16 19:21:12.805598 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805451    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="155749d1-fb68-4fb3-be12-28beb31c639b" containerName="main"
Sep 16 19:21:12.805598 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805492    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea52e50f-80e0-4beb-ac49-eace29207018" containerName="storage-initializer"
Sep 16 19:21:12.805598 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805502    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea52e50f-80e0-4beb-ac49-eace29207018" containerName="storage-initializer"
Sep 16 19:21:12.805598 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805525    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005dace5-f43e-468f-9240-00ed01738c70" containerName="main"
Sep 16 19:21:12.805598 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805544    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="005dace5-f43e-468f-9240-00ed01738c70" containerName="main"
Sep 16 19:21:12.805598 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805555    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155749d1-fb68-4fb3-be12-28beb31c639b" containerName="storage-initializer"
Sep 16 19:21:12.805598 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805562    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="155749d1-fb68-4fb3-be12-28beb31c639b" containerName="storage-initializer"
Sep 16 19:21:12.806013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805637    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="155749d1-fb68-4fb3-be12-28beb31c639b" containerName="main"
Sep 16 19:21:12.806013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805645    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea52e50f-80e0-4beb-ac49-eace29207018" containerName="main"
Sep 16 19:21:12.806013 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.805661    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="005dace5-f43e-468f-9240-00ed01738c70" containerName="main"
Sep 16 19:21:12.810326 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.810311    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:12.818457 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.818430    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-l4nkv\""
Sep 16 19:21:12.818608 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.818478    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 19:21:12.818754 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.818736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 16 19:21:12.819354 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.818987    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 19:21:12.820940 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.820919    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"]
Sep 16 19:21:12.922123 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.922093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:12.922292 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.922128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8549674-3306-4152-9a03-e8769f28fb0c-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:12.922292 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.922147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-home\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:12.922292 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.922255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tsj\" (UniqueName: \"kubernetes.io/projected/e8549674-3306-4152-9a03-e8769f28fb0c-kube-api-access-h9tsj\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:12.922457 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.922315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-model-cache\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:12.922457 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:12.922347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-dshm\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.023595 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.023566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.023736 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.023602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8549674-3306-4152-9a03-e8769f28fb0c-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.023736 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.023619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-home\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.023736 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.023665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tsj\" (UniqueName: \"kubernetes.io/projected/e8549674-3306-4152-9a03-e8769f28fb0c-kube-api-access-h9tsj\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.023736 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.023693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-model-cache\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.023736 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.023710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-dshm\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.024059 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.024037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.024115 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.024063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-home\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.024115 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.024100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-model-cache\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.026122 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.026100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-dshm\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.026509 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.026489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8549674-3306-4152-9a03-e8769f28fb0c-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.033353 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.033329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tsj\" (UniqueName: \"kubernetes.io/projected/e8549674-3306-4152-9a03-e8769f28fb0c-kube-api-access-h9tsj\") pod \"tri-v2-kserve-dcb554bf5-zlr4w\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.127906 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.127833    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:13.401234 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:13.401159    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"]
Sep 16 19:21:13.404715 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:21:13.404688    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8549674_3306_4152_9a03_e8769f28fb0c.slice/crio-5d4155b15231517b364dbeb1d0a93a23648523b6ea32455b7fad81d00dfb17d8 WatchSource:0}: Error finding container 5d4155b15231517b364dbeb1d0a93a23648523b6ea32455b7fad81d00dfb17d8: Status 404 returned error can't find the container with id 5d4155b15231517b364dbeb1d0a93a23648523b6ea32455b7fad81d00dfb17d8
Sep 16 19:21:14.247849 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:14.247811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w" event={"ID":"e8549674-3306-4152-9a03-e8769f28fb0c","Type":"ContainerStarted","Data":"104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8"}
Sep 16 19:21:14.248336 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:14.247883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w" event={"ID":"e8549674-3306-4152-9a03-e8769f28fb0c","Type":"ContainerStarted","Data":"5d4155b15231517b364dbeb1d0a93a23648523b6ea32455b7fad81d00dfb17d8"}
Sep 16 19:21:14.269240 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:14.269189    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w" podStartSLOduration=2.269175845 podStartE2EDuration="2.269175845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:14.265665512 +0000 UTC m=+2254.527342523" watchObservedRunningTime="2026-09-16 19:21:14.269175845 +0000 UTC m=+2254.530852883"
Sep 16 19:21:15.128470 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:15.128434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:22.890875 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:22.890838    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"]
Sep 16 19:21:22.893586 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:22.893568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:22.896133 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:22.896114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 16 19:21:22.906176 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:22.906152    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"]
Sep 16 19:21:23.010053 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.010015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q487j\" (UniqueName: \"kubernetes.io/projected/34b18801-7781-4bf6-b162-d1f2b9d5e972-kube-api-access-q487j\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.010244 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.010101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.010244 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.010129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b18801-7781-4bf6-b162-d1f2b9d5e972-tls-certs\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.010244 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.010150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-model-cache\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.010244 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.010234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-home\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.010414 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.010283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-dshm\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.111038 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.111000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.111038 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.111041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b18801-7781-4bf6-b162-d1f2b9d5e972-tls-certs\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.111272 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.111066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-model-cache\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.111272 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.111102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-home\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.111272 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.111260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-dshm\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.111440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.111286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q487j\" (UniqueName: \"kubernetes.io/projected/34b18801-7781-4bf6-b162-d1f2b9d5e972-kube-api-access-q487j\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.111500 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.111439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.111548 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.111494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-home\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.111548 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.111509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-model-cache\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.113661 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.113632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-dshm\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.113979 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.113957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b18801-7781-4bf6-b162-d1f2b9d5e972-tls-certs\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.119852 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.119835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q487j\" (UniqueName: \"kubernetes.io/projected/34b18801-7781-4bf6-b162-d1f2b9d5e972-kube-api-access-q487j\") pod \"tri-v3-kserve-798c88c66c-f84bv\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.128443 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.128417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:23.143847 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.143792    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:23.204677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.204652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:23.312697 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.312668    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:21:23.477413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:23.477365    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"]
Sep 16 19:21:23.479750 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:21:23.479724    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b18801_7781_4bf6_b162_d1f2b9d5e972.slice/crio-859fab08f937cdd74f9295bfb4c25427943e9c8454b1499b46a33ea043b9ae76 WatchSource:0}: Error finding container 859fab08f937cdd74f9295bfb4c25427943e9c8454b1499b46a33ea043b9ae76: Status 404 returned error can't find the container with id 859fab08f937cdd74f9295bfb4c25427943e9c8454b1499b46a33ea043b9ae76
Sep 16 19:21:24.308256 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:24.308216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv" event={"ID":"34b18801-7781-4bf6-b162-d1f2b9d5e972","Type":"ContainerStarted","Data":"8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8"}
Sep 16 19:21:24.308749 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:24.308282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv" event={"ID":"34b18801-7781-4bf6-b162-d1f2b9d5e972","Type":"ContainerStarted","Data":"859fab08f937cdd74f9295bfb4c25427943e9c8454b1499b46a33ea043b9ae76"}
Sep 16 19:21:24.337389 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:24.337340    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv" podStartSLOduration=2.337322677 podStartE2EDuration="2.337322677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:24.33584708 +0000 UTC m=+2264.597524094" watchObservedRunningTime="2026-09-16 19:21:24.337322677 +0000 UTC m=+2264.598999690"
Sep 16 19:21:25.205551 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:25.205516    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:33.204770 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:33.204734    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:33.217460 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:33.217431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:21:33.362594 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:21:33.362561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:22:27.787391 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.787351    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"]
Sep 16 19:22:27.790832 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.790814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.793440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.793413    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:22:27.794107 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.794076    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:22:27.794234 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.794206    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-c2nv5\""
Sep 16 19:22:27.794321 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.794300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:22:27.800659 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.800631    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"]
Sep 16 19:22:27.885093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.885051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4vz\" (UniqueName: \"kubernetes.io/projected/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-kube-api-access-lk4vz\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.885306 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.885115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.885306 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.885160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.885306 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.885181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.885306 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.885198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.885306 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.885229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.986055 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.986012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.986262 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.986093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4vz\" (UniqueName: \"kubernetes.io/projected/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-kube-api-access-lk4vz\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.986262 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.986132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.986262 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.986154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.986262 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.986181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.986262 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.986205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.986545 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.986532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.986591 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.986531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.986633 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.986602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.988764 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.988736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.989087 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.989070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:27.996334 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:27.996307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4vz\" (UniqueName: \"kubernetes.io/projected/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-kube-api-access-lk4vz\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:28.103390 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:28.103285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:28.388494 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:28.388465    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"]
Sep 16 19:22:28.390777 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:22:28.390749    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a7f0eb_2ee3_4cf3_89e1_05f23373ef7f.slice/crio-f2d5c2e126a260aebe97025a85b9e1d3513f472a9568c966bdf0326936b4b625 WatchSource:0}: Error finding container f2d5c2e126a260aebe97025a85b9e1d3513f472a9568c966bdf0326936b4b625: Status 404 returned error can't find the container with id f2d5c2e126a260aebe97025a85b9e1d3513f472a9568c966bdf0326936b4b625
Sep 16 19:22:28.615487 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:28.615450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p" event={"ID":"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f","Type":"ContainerStarted","Data":"72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742"}
Sep 16 19:22:28.615670 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:28.615501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p" event={"ID":"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f","Type":"ContainerStarted","Data":"f2d5c2e126a260aebe97025a85b9e1d3513f472a9568c966bdf0326936b4b625"}
Sep 16 19:22:28.635590 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:28.635526    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p" podStartSLOduration=1.635503981 podStartE2EDuration="1.635503981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:28.63430933 +0000 UTC m=+2328.895986345" watchObservedRunningTime="2026-09-16 19:22:28.635503981 +0000 UTC m=+2328.897180993"
Sep 16 19:22:29.104185 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:29.104144    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:38.103848 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:38.103798    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:38.116342 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:38.116317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:22:38.673796 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:22:38.673756    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:23:10.020603 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.020567    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"]
Sep 16 19:23:10.030371 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:23:10.030329    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:23:10.030371 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:23:10.030371 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:23:10.030371 ip-10-0-139-95 kubenswrapper[2580]:  > containerID="72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742" cmd=["/bin/sleep","15"]
Sep 16 19:23:10.030677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.030395    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p" message="" err=<
Sep 16 19:23:10.030677 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:23:10.030677 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:23:10.030677 ip-10-0-139-95 kubenswrapper[2580]:  >
Sep 16 19:23:10.030677 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:23:10.030411    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:23:10.030677 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:23:10.030677 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:23:10.030677 ip-10-0-139-95 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p" podUID="52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" containerName="main" containerID="cri-o://72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742"
Sep 16 19:23:10.030677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.030464    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p" podUID="52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" containerName="main" containerID="cri-o://72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742" gracePeriod=60
Sep 16 19:23:10.663304 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.663272    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p" podUID="52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:23:10.783683 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.783659    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:23:10.821557 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.821525    2580 generic.go:356] "Generic (PLEG): container finished" podID="52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" containerID="72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742" exitCode=0
Sep 16 19:23:10.821716 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.821573    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"
Sep 16 19:23:10.821716 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.821614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p" event={"ID":"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f","Type":"ContainerDied","Data":"72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742"}
Sep 16 19:23:10.821716 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.821672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p" event={"ID":"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f","Type":"ContainerDied","Data":"f2d5c2e126a260aebe97025a85b9e1d3513f472a9568c966bdf0326936b4b625"}
Sep 16 19:23:10.821716 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.821700    2580 scope.go:117] "RemoveContainer" containerID="72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742"
Sep 16 19:23:10.830769 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.830745    2580 scope.go:117] "RemoveContainer" containerID="72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742"
Sep 16 19:23:10.831083 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:23:10.831060    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742\": container with ID starting with 72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742 not found: ID does not exist" containerID="72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742"
Sep 16 19:23:10.831196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.831090    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742"} err="failed to get container status \"72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742\": rpc error: code = NotFound desc = could not find container \"72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742\": container with ID starting with 72cdc6f8de32ac0e642d97a1070915c3889137d512f0aea191dae5d27ec49742 not found: ID does not exist"
Sep 16 19:23:10.873337 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873306    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-dshm\") pod \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") "
Sep 16 19:23:10.873528 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tmp-dir\") pod \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") "
Sep 16 19:23:10.873528 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tls-certs\") pod \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") "
Sep 16 19:23:10.873528 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-model-cache\") pod \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") "
Sep 16 19:23:10.873528 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-home\") pod \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") "
Sep 16 19:23:10.873765 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873579    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk4vz\" (UniqueName: \"kubernetes.io/projected/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-kube-api-access-lk4vz\") pod \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\" (UID: \"52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f\") "
Sep 16 19:23:10.873765 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" (UID: "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:10.873871 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" (UID: "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:10.873871 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873845    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-home" (OuterVolumeSpecName: "home") pod "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" (UID: "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:10.874010 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873946    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:10.874010 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.873967    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:10.875860 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.875830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-dshm" (OuterVolumeSpecName: "dshm") pod "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" (UID: "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:10.875987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.875883    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" (UID: "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:10.875987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.875885    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-kube-api-access-lk4vz" (OuterVolumeSpecName: "kube-api-access-lk4vz") pod "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" (UID: "52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f"). InnerVolumeSpecName "kube-api-access-lk4vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:10.975375 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.975341    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:10.975375 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.975372    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:10.975375 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.975383    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:10.975627 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:10.975393    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4vz\" (UniqueName: \"kubernetes.io/projected/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f-kube-api-access-lk4vz\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:11.146435 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:11.146404    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"]
Sep 16 19:23:11.151102 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:11.151078    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jq24p"]
Sep 16 19:23:12.275990 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:12.275955    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" path="/var/lib/kubelet/pods/52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f/volumes"
Sep 16 19:23:40.588948 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:40.588911    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"]
Sep 16 19:23:41.430324 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:41.430288    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"]
Sep 16 19:23:55.608265 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.608206    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w" podUID="e8549674-3306-4152-9a03-e8769f28fb0c" containerName="main" containerID="cri-o://104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8" gracePeriod=45
Sep 16 19:23:55.853697 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.853671    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:23:55.955172 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955140    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-tmp-dir\") pod \"e8549674-3306-4152-9a03-e8769f28fb0c\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") "
Sep 16 19:23:55.955497 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-home\") pod \"e8549674-3306-4152-9a03-e8769f28fb0c\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") "
Sep 16 19:23:55.955497 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9tsj\" (UniqueName: \"kubernetes.io/projected/e8549674-3306-4152-9a03-e8769f28fb0c-kube-api-access-h9tsj\") pod \"e8549674-3306-4152-9a03-e8769f28fb0c\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") "
Sep 16 19:23:55.955497 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8549674-3306-4152-9a03-e8769f28fb0c-tls-certs\") pod \"e8549674-3306-4152-9a03-e8769f28fb0c\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") "
Sep 16 19:23:55.955497 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955306    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-model-cache\") pod \"e8549674-3306-4152-9a03-e8769f28fb0c\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") "
Sep 16 19:23:55.955497 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-dshm\") pod \"e8549674-3306-4152-9a03-e8769f28fb0c\" (UID: \"e8549674-3306-4152-9a03-e8769f28fb0c\") "
Sep 16 19:23:55.955497 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8549674-3306-4152-9a03-e8769f28fb0c" (UID: "e8549674-3306-4152-9a03-e8769f28fb0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:55.955497 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-home" (OuterVolumeSpecName: "home") pod "e8549674-3306-4152-9a03-e8769f28fb0c" (UID: "e8549674-3306-4152-9a03-e8769f28fb0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:55.955919 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955582    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:55.955919 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955599    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:55.956198 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.955950    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8549674-3306-4152-9a03-e8769f28fb0c" (UID: "e8549674-3306-4152-9a03-e8769f28fb0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:55.957822 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.957798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8549674-3306-4152-9a03-e8769f28fb0c-kube-api-access-h9tsj" (OuterVolumeSpecName: "kube-api-access-h9tsj") pod "e8549674-3306-4152-9a03-e8769f28fb0c" (UID: "e8549674-3306-4152-9a03-e8769f28fb0c"). InnerVolumeSpecName "kube-api-access-h9tsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:55.957822 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.957800    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8549674-3306-4152-9a03-e8769f28fb0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8549674-3306-4152-9a03-e8769f28fb0c" (UID: "e8549674-3306-4152-9a03-e8769f28fb0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:55.958034 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:55.958018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-dshm" (OuterVolumeSpecName: "dshm") pod "e8549674-3306-4152-9a03-e8769f28fb0c" (UID: "e8549674-3306-4152-9a03-e8769f28fb0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:56.033129 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.033101    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8549674-3306-4152-9a03-e8769f28fb0c" containerID="104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8" exitCode=0
Sep 16 19:23:56.033266 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.033146    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"
Sep 16 19:23:56.033266 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.033185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w" event={"ID":"e8549674-3306-4152-9a03-e8769f28fb0c","Type":"ContainerDied","Data":"104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8"}
Sep 16 19:23:56.033266 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.033240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w" event={"ID":"e8549674-3306-4152-9a03-e8769f28fb0c","Type":"ContainerDied","Data":"5d4155b15231517b364dbeb1d0a93a23648523b6ea32455b7fad81d00dfb17d8"}
Sep 16 19:23:56.033266 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.033263    2580 scope.go:117] "RemoveContainer" containerID="104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8"
Sep 16 19:23:56.042225 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.042210    2580 scope.go:117] "RemoveContainer" containerID="104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8"
Sep 16 19:23:56.042508 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:23:56.042490    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8\": container with ID starting with 104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8 not found: ID does not exist" containerID="104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8"
Sep 16 19:23:56.042557 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.042516    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8"} err="failed to get container status \"104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8\": rpc error: code = NotFound desc = could not find container \"104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8\": container with ID starting with 104b74ff095e12146fb8ab86d9c7a97e0928e27852d018e2b1175b1179e68cd8 not found: ID does not exist"
Sep 16 19:23:56.054794 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.054772    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"]
Sep 16 19:23:56.056662 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.056640    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9tsj\" (UniqueName: \"kubernetes.io/projected/e8549674-3306-4152-9a03-e8769f28fb0c-kube-api-access-h9tsj\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:56.056662 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.056662    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8549674-3306-4152-9a03-e8769f28fb0c-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:56.056785 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.056672    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:56.056785 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.056680    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8549674-3306-4152-9a03-e8769f28fb0c-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:56.062413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.062394    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-zlr4w"]
Sep 16 19:23:56.276048 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.275980    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8549674-3306-4152-9a03-e8769f28fb0c" path="/var/lib/kubelet/pods/e8549674-3306-4152-9a03-e8769f28fb0c/volumes"
Sep 16 19:23:56.449649 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.449612    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv" podUID="34b18801-7781-4bf6-b162-d1f2b9d5e972" containerName="main" containerID="cri-o://8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8" gracePeriod=45
Sep 16 19:23:56.696316 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.696294    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:23:56.761525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.761468    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b18801-7781-4bf6-b162-d1f2b9d5e972-tls-certs\") pod \"34b18801-7781-4bf6-b162-d1f2b9d5e972\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") "
Sep 16 19:23:56.761677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.761532    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-dshm\") pod \"34b18801-7781-4bf6-b162-d1f2b9d5e972\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") "
Sep 16 19:23:56.761677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.761586    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q487j\" (UniqueName: \"kubernetes.io/projected/34b18801-7781-4bf6-b162-d1f2b9d5e972-kube-api-access-q487j\") pod \"34b18801-7781-4bf6-b162-d1f2b9d5e972\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") "
Sep 16 19:23:56.761677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.761610    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-model-cache\") pod \"34b18801-7781-4bf6-b162-d1f2b9d5e972\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") "
Sep 16 19:23:56.761677 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.761643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-tmp-dir\") pod \"34b18801-7781-4bf6-b162-d1f2b9d5e972\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") "
Sep 16 19:23:56.761908 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.761691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-home\") pod \"34b18801-7781-4bf6-b162-d1f2b9d5e972\" (UID: \"34b18801-7781-4bf6-b162-d1f2b9d5e972\") "
Sep 16 19:23:56.762196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.762169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-home" (OuterVolumeSpecName: "home") pod "34b18801-7781-4bf6-b162-d1f2b9d5e972" (UID: "34b18801-7781-4bf6-b162-d1f2b9d5e972"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:56.762420 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.762400    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34b18801-7781-4bf6-b162-d1f2b9d5e972" (UID: "34b18801-7781-4bf6-b162-d1f2b9d5e972"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:56.762500 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.762465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-model-cache" (OuterVolumeSpecName: "model-cache") pod "34b18801-7781-4bf6-b162-d1f2b9d5e972" (UID: "34b18801-7781-4bf6-b162-d1f2b9d5e972"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:56.764194 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.764170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b18801-7781-4bf6-b162-d1f2b9d5e972-kube-api-access-q487j" (OuterVolumeSpecName: "kube-api-access-q487j") pod "34b18801-7781-4bf6-b162-d1f2b9d5e972" (UID: "34b18801-7781-4bf6-b162-d1f2b9d5e972"). InnerVolumeSpecName "kube-api-access-q487j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:56.764398 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.764379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-dshm" (OuterVolumeSpecName: "dshm") pod "34b18801-7781-4bf6-b162-d1f2b9d5e972" (UID: "34b18801-7781-4bf6-b162-d1f2b9d5e972"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:56.764454 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.764391    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b18801-7781-4bf6-b162-d1f2b9d5e972-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34b18801-7781-4bf6-b162-d1f2b9d5e972" (UID: "34b18801-7781-4bf6-b162-d1f2b9d5e972"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:56.862554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.862480    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b18801-7781-4bf6-b162-d1f2b9d5e972-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:56.862554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.862510    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:56.862554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.862523    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q487j\" (UniqueName: \"kubernetes.io/projected/34b18801-7781-4bf6-b162-d1f2b9d5e972-kube-api-access-q487j\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:56.862554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.862536    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:56.862554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.862547    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:56.862554 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:56.862558    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34b18801-7781-4bf6-b162-d1f2b9d5e972-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:23:57.040377 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:57.040344    2580 generic.go:356] "Generic (PLEG): container finished" podID="34b18801-7781-4bf6-b162-d1f2b9d5e972" containerID="8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8" exitCode=0
Sep 16 19:23:57.040569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:57.040388    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"
Sep 16 19:23:57.040569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:57.040435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv" event={"ID":"34b18801-7781-4bf6-b162-d1f2b9d5e972","Type":"ContainerDied","Data":"8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8"}
Sep 16 19:23:57.040569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:57.040506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv" event={"ID":"34b18801-7781-4bf6-b162-d1f2b9d5e972","Type":"ContainerDied","Data":"859fab08f937cdd74f9295bfb4c25427943e9c8454b1499b46a33ea043b9ae76"}
Sep 16 19:23:57.040569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:57.040536    2580 scope.go:117] "RemoveContainer" containerID="8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8"
Sep 16 19:23:57.049820 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:57.049803    2580 scope.go:117] "RemoveContainer" containerID="8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8"
Sep 16 19:23:57.050228 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:23:57.050208    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8\": container with ID starting with 8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8 not found: ID does not exist" containerID="8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8"
Sep 16 19:23:57.050301 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:57.050234    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8"} err="failed to get container status \"8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8\": rpc error: code = NotFound desc = could not find container \"8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8\": container with ID starting with 8148b014da0a40479e4f7bedb542f7eedafc894e9b8cd87df3b8d17e514a2fe8 not found: ID does not exist"
Sep 16 19:23:57.062803 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:57.062777    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"]
Sep 16 19:23:57.071537 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:57.071512    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-f84bv"]
Sep 16 19:23:58.275614 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:23:58.275583    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34b18801-7781-4bf6-b162-d1f2b9d5e972" path="/var/lib/kubelet/pods/34b18801-7781-4bf6-b162-d1f2b9d5e972/volumes"
Sep 16 19:24:13.970137 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970059    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"]
Sep 16 19:24:13.970804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970546    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" containerName="main"
Sep 16 19:24:13.970804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970559    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" containerName="main"
Sep 16 19:24:13.970804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970573    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8549674-3306-4152-9a03-e8769f28fb0c" containerName="main"
Sep 16 19:24:13.970804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970582    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8549674-3306-4152-9a03-e8769f28fb0c" containerName="main"
Sep 16 19:24:13.970804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970597    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34b18801-7781-4bf6-b162-d1f2b9d5e972" containerName="main"
Sep 16 19:24:13.970804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970608    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b18801-7781-4bf6-b162-d1f2b9d5e972" containerName="main"
Sep 16 19:24:13.970804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970710    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34b18801-7781-4bf6-b162-d1f2b9d5e972" containerName="main"
Sep 16 19:24:13.970804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970721    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8549674-3306-4152-9a03-e8769f28fb0c" containerName="main"
Sep 16 19:24:13.970804 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.970729    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a7f0eb-2ee3-4cf3-89e1-05f23373ef7f" containerName="main"
Sep 16 19:24:13.975291 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.975267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:24:13.982557 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.982533    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:24:13.983420 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.983401    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-cjx6p\""
Sep 16 19:24:13.983525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.983401    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-zpg8q\""
Sep 16 19:24:13.983525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.983407    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:24:13.983525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.983405    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:24:13.990207 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:13.990182    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"]
Sep 16 19:24:14.111232 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:14.111198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40db9665-2b99-476f-a0bd-df017c557c76-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8\" (UID: \"40db9665-2b99-476f-a0bd-df017c557c76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:24:14.111376 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:14.111248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkdp\" (UniqueName: \"kubernetes.io/projected/40db9665-2b99-476f-a0bd-df017c557c76-kube-api-access-fpkdp\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8\" (UID: \"40db9665-2b99-476f-a0bd-df017c557c76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:24:14.212197 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:14.212170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40db9665-2b99-476f-a0bd-df017c557c76-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8\" (UID: \"40db9665-2b99-476f-a0bd-df017c557c76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:24:14.212334 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:14.212216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkdp\" (UniqueName: \"kubernetes.io/projected/40db9665-2b99-476f-a0bd-df017c557c76-kube-api-access-fpkdp\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8\" (UID: \"40db9665-2b99-476f-a0bd-df017c557c76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:24:14.214811 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:14.214783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40db9665-2b99-476f-a0bd-df017c557c76-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8\" (UID: \"40db9665-2b99-476f-a0bd-df017c557c76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:24:14.221384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:14.221323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkdp\" (UniqueName: \"kubernetes.io/projected/40db9665-2b99-476f-a0bd-df017c557c76-kube-api-access-fpkdp\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8\" (UID: \"40db9665-2b99-476f-a0bd-df017c557c76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:24:14.285822 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:14.285803    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:24:14.576249 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:14.576171    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"]
Sep 16 19:24:14.579122 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:24:14.579086    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40db9665_2b99_476f_a0bd_df017c557c76.slice/crio-1403959cc372228975af7ae2e764eefa2eecb919ffd7c4e7d70f0ed03b622143 WatchSource:0}: Error finding container 1403959cc372228975af7ae2e764eefa2eecb919ffd7c4e7d70f0ed03b622143: Status 404 returned error can't find the container with id 1403959cc372228975af7ae2e764eefa2eecb919ffd7c4e7d70f0ed03b622143
Sep 16 19:24:15.131573 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:15.131537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" event={"ID":"40db9665-2b99-476f-a0bd-df017c557c76","Type":"ContainerStarted","Data":"64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b"}
Sep 16 19:24:15.132090 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:15.131596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" event={"ID":"40db9665-2b99-476f-a0bd-df017c557c76","Type":"ContainerStarted","Data":"1403959cc372228975af7ae2e764eefa2eecb919ffd7c4e7d70f0ed03b622143"}
Sep 16 19:24:15.132090 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:15.131628    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:24:15.152727 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:15.152672    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" podStartSLOduration=2.15265436 podStartE2EDuration="2.15265436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:15.151608936 +0000 UTC m=+2435.413285951" watchObservedRunningTime="2026-09-16 19:24:15.15265436 +0000 UTC m=+2435.414331371"
Sep 16 19:24:46.138030 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:24:46.137997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:26:34.415104 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.415067    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"]
Sep 16 19:26:34.418760 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.418736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.421201 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.421176    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 19:26:34.421850 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.421832    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 19:26:34.422047 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.422009    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 19:26:34.422122 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.422066    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-62x7b\""
Sep 16 19:26:34.427606 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.427585    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"]
Sep 16 19:26:34.512996 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.512958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.513196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.513005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-home\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.513196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.513114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-dshm\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.513196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.513147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.513196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.513173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjxt\" (UniqueName: \"kubernetes.io/projected/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-kube-api-access-5wjxt\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.513385 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.513262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.614528 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.614486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-dshm\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.614528 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.614531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.614807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.614563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjxt\" (UniqueName: \"kubernetes.io/projected/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-kube-api-access-5wjxt\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.614807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.614611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.614807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.614652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.614807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.614681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-home\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.615101 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.615073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.615162 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.615101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-home\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.615201 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.615157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.617066 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.617047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-dshm\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.617456 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.617437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.622954 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.622932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjxt\" (UniqueName: \"kubernetes.io/projected/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-kube-api-access-5wjxt\") pod \"delw-v1-kserve-6cf8cbb847-kvkqh\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:34.730978 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:34.730939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:35.012334 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:35.012257    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"]
Sep 16 19:26:35.016525 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:26:35.016491    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7f2d6e_3d85_4c69_bd4e_c78fd051f9cb.slice/crio-68b942db601e6885ef4db10eeaee631a5607c6507df93921e2656047663c8fb2 WatchSource:0}: Error finding container 68b942db601e6885ef4db10eeaee631a5607c6507df93921e2656047663c8fb2: Status 404 returned error can't find the container with id 68b942db601e6885ef4db10eeaee631a5607c6507df93921e2656047663c8fb2
Sep 16 19:26:35.789575 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:35.789540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh" event={"ID":"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb","Type":"ContainerStarted","Data":"77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50"}
Sep 16 19:26:35.790010 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:35.789592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh" event={"ID":"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb","Type":"ContainerStarted","Data":"68b942db601e6885ef4db10eeaee631a5607c6507df93921e2656047663c8fb2"}
Sep 16 19:26:35.810826 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:35.810770    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh" podStartSLOduration=1.810753154 podStartE2EDuration="1.810753154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:35.81014663 +0000 UTC m=+2576.071823653" watchObservedRunningTime="2026-09-16 19:26:35.810753154 +0000 UTC m=+2576.072430167"
Sep 16 19:26:36.731668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:36.731625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:44.732063 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:44.732017    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:44.744569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:44.744540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:26:44.843688 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:26:44.843659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:27:00.709459 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:00.709423    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"]
Sep 16 19:27:06.137864 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:06.137828    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" podUID="40db9665-2b99-476f-a0bd-df017c557c76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:15.730074 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:15.730007    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" podUID="40db9665-2b99-476f-a0bd-df017c557c76" containerName="main" containerID="cri-o://64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b" gracePeriod=45
Sep 16 19:27:16.137687 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:16.137605    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" podUID="40db9665-2b99-476f-a0bd-df017c557c76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:26.137764 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:26.137726    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" podUID="40db9665-2b99-476f-a0bd-df017c557c76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:26.138347 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:26.137847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:27:36.137828 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:36.137780    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" podUID="40db9665-2b99-476f-a0bd-df017c557c76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:45.983552 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:45.983526    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:27:46.126220 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.126127    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpkdp\" (UniqueName: \"kubernetes.io/projected/40db9665-2b99-476f-a0bd-df017c557c76-kube-api-access-fpkdp\") pod \"40db9665-2b99-476f-a0bd-df017c557c76\" (UID: \"40db9665-2b99-476f-a0bd-df017c557c76\") "
Sep 16 19:27:46.126220 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.126179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40db9665-2b99-476f-a0bd-df017c557c76-tls-certs\") pod \"40db9665-2b99-476f-a0bd-df017c557c76\" (UID: \"40db9665-2b99-476f-a0bd-df017c557c76\") "
Sep 16 19:27:46.128726 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.128699    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40db9665-2b99-476f-a0bd-df017c557c76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40db9665-2b99-476f-a0bd-df017c557c76" (UID: "40db9665-2b99-476f-a0bd-df017c557c76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:46.128726 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.128706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40db9665-2b99-476f-a0bd-df017c557c76-kube-api-access-fpkdp" (OuterVolumeSpecName: "kube-api-access-fpkdp") pod "40db9665-2b99-476f-a0bd-df017c557c76" (UID: "40db9665-2b99-476f-a0bd-df017c557c76"). InnerVolumeSpecName "kube-api-access-fpkdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:46.132170 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.132141    2580 generic.go:356] "Generic (PLEG): container finished" podID="40db9665-2b99-476f-a0bd-df017c557c76" containerID="64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b" exitCode=0
Sep 16 19:27:46.132283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.132186    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"
Sep 16 19:27:46.132283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.132219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" event={"ID":"40db9665-2b99-476f-a0bd-df017c557c76","Type":"ContainerDied","Data":"64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b"}
Sep 16 19:27:46.132283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.132279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8" event={"ID":"40db9665-2b99-476f-a0bd-df017c557c76","Type":"ContainerDied","Data":"1403959cc372228975af7ae2e764eefa2eecb919ffd7c4e7d70f0ed03b622143"}
Sep 16 19:27:46.132458 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.132310    2580 scope.go:117] "RemoveContainer" containerID="64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b"
Sep 16 19:27:46.146171 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.146149    2580 scope.go:117] "RemoveContainer" containerID="64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b"
Sep 16 19:27:46.146439 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:27:46.146418    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b\": container with ID starting with 64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b not found: ID does not exist" containerID="64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b"
Sep 16 19:27:46.146511 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.146451    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b"} err="failed to get container status \"64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b\": rpc error: code = NotFound desc = could not find container \"64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b\": container with ID starting with 64fdb466addd833eeb930cb1df035abf26c2e66a2a37c9f8388af8ebbe7e846b not found: ID does not exist"
Sep 16 19:27:46.157778 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.157755    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"]
Sep 16 19:27:46.161859 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.161838    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8qfw8"]
Sep 16 19:27:46.227646 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.227608    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpkdp\" (UniqueName: \"kubernetes.io/projected/40db9665-2b99-476f-a0bd-df017c557c76-kube-api-access-fpkdp\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:27:46.227646 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.227641    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40db9665-2b99-476f-a0bd-df017c557c76-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:27:46.275976 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:27:46.275942    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40db9665-2b99-476f-a0bd-df017c557c76" path="/var/lib/kubelet/pods/40db9665-2b99-476f-a0bd-df017c557c76/volumes"
Sep 16 19:28:05.606461 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.606431    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"]
Sep 16 19:28:05.606817 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.606792    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40db9665-2b99-476f-a0bd-df017c557c76" containerName="main"
Sep 16 19:28:05.606817 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.606803    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40db9665-2b99-476f-a0bd-df017c557c76" containerName="main"
Sep 16 19:28:05.606913 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.606859    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40db9665-2b99-476f-a0bd-df017c557c76" containerName="main"
Sep 16 19:28:05.609826 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.609807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.612511 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.612485    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:28:05.612624 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.612506    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:28:05.612624 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.612545    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-nj5jj\""
Sep 16 19:28:05.612624 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.612500    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:28:05.621563 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.621541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"]
Sep 16 19:28:05.696059 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.696034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.696207 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.696084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.696207 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.696103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.696207 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.696150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.696207 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.696182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2698j\" (UniqueName: \"kubernetes.io/projected/f1140a7b-0ff2-4e98-9340-70aa893ac5db-kube-api-access-2698j\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.696372 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.696233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.797627 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.797600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.797758 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.797629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.797758 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.797659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.797758 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.797703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2698j\" (UniqueName: \"kubernetes.io/projected/f1140a7b-0ff2-4e98-9340-70aa893ac5db-kube-api-access-2698j\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.797758 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.797748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.798021 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.797801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.798103 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.798081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.798163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.798147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.798230 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.798210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.800086 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.800069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.800427 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.800412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.806668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.806650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2698j\" (UniqueName: \"kubernetes.io/projected/f1140a7b-0ff2-4e98-9340-70aa893ac5db-kube-api-access-2698j\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:05.920483 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:05.920457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:06.199260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:06.199237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"]
Sep 16 19:28:06.201574 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:28:06.201542    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1140a7b_0ff2_4e98_9340_70aa893ac5db.slice/crio-5a9c9bb31a60a2a5401cfb1c4b52eff877e4d4f79025369e9deb1cf11a088bb6 WatchSource:0}: Error finding container 5a9c9bb31a60a2a5401cfb1c4b52eff877e4d4f79025369e9deb1cf11a088bb6: Status 404 returned error can't find the container with id 5a9c9bb31a60a2a5401cfb1c4b52eff877e4d4f79025369e9deb1cf11a088bb6
Sep 16 19:28:06.227631 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:06.227600    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp" event={"ID":"f1140a7b-0ff2-4e98-9340-70aa893ac5db","Type":"ContainerStarted","Data":"5a9c9bb31a60a2a5401cfb1c4b52eff877e4d4f79025369e9deb1cf11a088bb6"}
Sep 16 19:28:07.234484 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:07.234447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp" event={"ID":"f1140a7b-0ff2-4e98-9340-70aa893ac5db","Type":"ContainerStarted","Data":"44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1"}
Sep 16 19:28:07.268222 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:07.268171    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp" podStartSLOduration=2.268154034 podStartE2EDuration="2.268154034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:07.266764312 +0000 UTC m=+2667.528441324" watchObservedRunningTime="2026-09-16 19:28:07.268154034 +0000 UTC m=+2667.529831093"
Sep 16 19:28:07.921327 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:07.921294    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:14.844819 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:14.844787    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"]
Sep 16 19:28:15.921455 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:15.921411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:15.933844 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:15.933814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:16.288996 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:16.288879    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:28:29.864914 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:29.864857    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh" podUID="8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" containerName="main" containerID="cri-o://77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50" gracePeriod=45
Sep 16 19:28:30.105905 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.105864    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:28:30.205860 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.205823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wjxt\" (UniqueName: \"kubernetes.io/projected/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-kube-api-access-5wjxt\") pod \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") "
Sep 16 19:28:30.206080 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.205921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tls-certs\") pod \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") "
Sep 16 19:28:30.206080 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.205950    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-model-cache\") pod \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") "
Sep 16 19:28:30.206080 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.206015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-home\") pod \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") "
Sep 16 19:28:30.206080 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.206049    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tmp-dir\") pod \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") "
Sep 16 19:28:30.206080 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.206074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-dshm\") pod \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\" (UID: \"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb\") "
Sep 16 19:28:30.206354 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.206267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" (UID: "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:30.206354 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.206286    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-home" (OuterVolumeSpecName: "home") pod "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" (UID: "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:30.206354 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.206313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" (UID: "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:30.208414 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.208389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-dshm" (OuterVolumeSpecName: "dshm") pod "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" (UID: "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:30.208531 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.208392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" (UID: "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:30.208531 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.208432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-kube-api-access-5wjxt" (OuterVolumeSpecName: "kube-api-access-5wjxt") pod "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" (UID: "8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb"). InnerVolumeSpecName "kube-api-access-5wjxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:30.307212 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.307179    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:28:30.307212 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.307206    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:28:30.307212 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.307214    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:28:30.307421 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.307224    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjxt\" (UniqueName: \"kubernetes.io/projected/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-kube-api-access-5wjxt\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:28:30.307421 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.307234    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:28:30.307421 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.307243    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:28:30.344922 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.344857    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" containerID="77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50" exitCode=0
Sep 16 19:28:30.345093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.344931    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"
Sep 16 19:28:30.345093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.344939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh" event={"ID":"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb","Type":"ContainerDied","Data":"77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50"}
Sep 16 19:28:30.345093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.344996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh" event={"ID":"8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb","Type":"ContainerDied","Data":"68b942db601e6885ef4db10eeaee631a5607c6507df93921e2656047663c8fb2"}
Sep 16 19:28:30.345093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.345021    2580 scope.go:117] "RemoveContainer" containerID="77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50"
Sep 16 19:28:30.353875 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.353857    2580 scope.go:117] "RemoveContainer" containerID="77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50"
Sep 16 19:28:30.354188 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:28:30.354169    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50\": container with ID starting with 77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50 not found: ID does not exist" containerID="77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50"
Sep 16 19:28:30.354248 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.354198    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50"} err="failed to get container status \"77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50\": rpc error: code = NotFound desc = could not find container \"77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50\": container with ID starting with 77419cade52dc4184e8045e0aa424372cc3e647ccf9827e759022c1f2f3bdf50 not found: ID does not exist"
Sep 16 19:28:30.361592 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.361570    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"]
Sep 16 19:28:30.367701 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:30.367680    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kvkqh"]
Sep 16 19:28:32.276227 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:28:32.276143    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" path="/var/lib/kubelet/pods/8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb/volumes"
Sep 16 19:31:38.658034 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:38.657998    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"]
Sep 16 19:31:38.667385 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:31:38.667352    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:31:38.667385 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:31:38.667385 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:31:38.667385 ip-10-0-139-95 kubenswrapper[2580]:  > containerID="44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1" cmd=["/bin/sleep","15"]
Sep 16 19:31:38.667708 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:38.667404    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp" message="" err=<
Sep 16 19:31:38.667708 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:31:38.667708 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:31:38.667708 ip-10-0-139-95 kubenswrapper[2580]:  >
Sep 16 19:31:38.667708 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:31:38.667416    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:31:38.667708 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:31:38.667708 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 19:31:38.667708 ip-10-0-139-95 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp" podUID="f1140a7b-0ff2-4e98-9340-70aa893ac5db" containerName="main" containerID="cri-o://44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1"
Sep 16 19:31:38.667708 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:38.667450    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp" podUID="f1140a7b-0ff2-4e98-9340-70aa893ac5db" containerName="main" containerID="cri-o://44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1" gracePeriod=60
Sep 16 19:31:39.010283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.010260    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:31:39.164530 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.164496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tls-certs\") pod \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") "
Sep 16 19:31:39.164530 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.164536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-home\") pod \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") "
Sep 16 19:31:39.164790 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.164603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-dshm\") pod \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") "
Sep 16 19:31:39.164790 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.164641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tmp-dir\") pod \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") "
Sep 16 19:31:39.164790 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.164663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2698j\" (UniqueName: \"kubernetes.io/projected/f1140a7b-0ff2-4e98-9340-70aa893ac5db-kube-api-access-2698j\") pod \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") "
Sep 16 19:31:39.164790 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.164703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-model-cache\") pod \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\" (UID: \"f1140a7b-0ff2-4e98-9340-70aa893ac5db\") "
Sep 16 19:31:39.165042 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.164886    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-home" (OuterVolumeSpecName: "home") pod "f1140a7b-0ff2-4e98-9340-70aa893ac5db" (UID: "f1140a7b-0ff2-4e98-9340-70aa893ac5db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:39.165103 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.165058    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1140a7b-0ff2-4e98-9340-70aa893ac5db" (UID: "f1140a7b-0ff2-4e98-9340-70aa893ac5db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:39.165151 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.165083    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:31:39.165151 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.165114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1140a7b-0ff2-4e98-9340-70aa893ac5db" (UID: "f1140a7b-0ff2-4e98-9340-70aa893ac5db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:39.167169 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.167138    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1140a7b-0ff2-4e98-9340-70aa893ac5db-kube-api-access-2698j" (OuterVolumeSpecName: "kube-api-access-2698j") pod "f1140a7b-0ff2-4e98-9340-70aa893ac5db" (UID: "f1140a7b-0ff2-4e98-9340-70aa893ac5db"). InnerVolumeSpecName "kube-api-access-2698j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:39.167169 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.167144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-dshm" (OuterVolumeSpecName: "dshm") pod "f1140a7b-0ff2-4e98-9340-70aa893ac5db" (UID: "f1140a7b-0ff2-4e98-9340-70aa893ac5db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:39.167337 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.167191    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1140a7b-0ff2-4e98-9340-70aa893ac5db" (UID: "f1140a7b-0ff2-4e98-9340-70aa893ac5db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:39.237016 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.236941    2580 generic.go:356] "Generic (PLEG): container finished" podID="f1140a7b-0ff2-4e98-9340-70aa893ac5db" containerID="44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1" exitCode=0
Sep 16 19:31:39.237016 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.236975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp" event={"ID":"f1140a7b-0ff2-4e98-9340-70aa893ac5db","Type":"ContainerDied","Data":"44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1"}
Sep 16 19:31:39.237016 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.236996    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"
Sep 16 19:31:39.237016 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.237016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp" event={"ID":"f1140a7b-0ff2-4e98-9340-70aa893ac5db","Type":"ContainerDied","Data":"5a9c9bb31a60a2a5401cfb1c4b52eff877e4d4f79025369e9deb1cf11a088bb6"}
Sep 16 19:31:39.237295 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.237036    2580 scope.go:117] "RemoveContainer" containerID="44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1"
Sep 16 19:31:39.247372 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.247355    2580 scope.go:117] "RemoveContainer" containerID="44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1"
Sep 16 19:31:39.247629 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:31:39.247615    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1\": container with ID starting with 44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1 not found: ID does not exist" containerID="44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1"
Sep 16 19:31:39.247710 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.247637    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1"} err="failed to get container status \"44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1\": rpc error: code = NotFound desc = could not find container \"44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1\": container with ID starting with 44c88c6b8832715170450dffa59c2125c7517a555dcfcbea4b946e8641506ef1 not found: ID does not exist"
Sep 16 19:31:39.267101 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.263252    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"]
Sep 16 19:31:39.267868 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.267843    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:31:39.267996 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.267874    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2698j\" (UniqueName: \"kubernetes.io/projected/f1140a7b-0ff2-4e98-9340-70aa893ac5db-kube-api-access-2698j\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:31:39.267996 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.267914    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:31:39.267996 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.267929    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1140a7b-0ff2-4e98-9340-70aa893ac5db-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:31:39.267996 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.267941    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1140a7b-0ff2-4e98-9340-70aa893ac5db-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:31:39.268237 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:39.268221    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796988kkp"]
Sep 16 19:31:40.275725 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:31:40.275693    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1140a7b-0ff2-4e98-9340-70aa893ac5db" path="/var/lib/kubelet/pods/f1140a7b-0ff2-4e98-9340-70aa893ac5db/volumes"
Sep 16 19:33:12.127398 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.127324    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"]
Sep 16 19:33:12.127833 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.127678    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1140a7b-0ff2-4e98-9340-70aa893ac5db" containerName="main"
Sep 16 19:33:12.127833 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.127690    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1140a7b-0ff2-4e98-9340-70aa893ac5db" containerName="main"
Sep 16 19:33:12.127833 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.127699    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" containerName="main"
Sep 16 19:33:12.127833 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.127704    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" containerName="main"
Sep 16 19:33:12.127833 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.127765    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7f2d6e-3d85-4c69-bd4e-c78fd051f9cb" containerName="main"
Sep 16 19:33:12.127833 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.127773    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1140a7b-0ff2-4e98-9340-70aa893ac5db" containerName="main"
Sep 16 19:33:12.130933 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.130872    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.133175 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.133152    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:33:12.133288 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.133207    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-5vv2l\""
Sep 16 19:33:12.133508 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.133482    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:33:12.133689 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.133676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:33:12.134736 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.134716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 19:33:12.134838 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.134820    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:33:12.146194 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.146175    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"]
Sep 16 19:33:12.219678 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.219647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lks25\" (UniqueName: \"kubernetes.io/projected/8280243e-6c6c-4a19-8594-284b8f80e708-kube-api-access-lks25\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.219827 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.219707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8280243e-6c6c-4a19-8594-284b8f80e708-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.219827 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.219764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.219827 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.219817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.219992 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.219839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.219992 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.219869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.219992 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.219886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8280243e-6c6c-4a19-8594-284b8f80e708-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.219992 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.219964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321314 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321314 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8280243e-6c6c-4a19-8594-284b8f80e708-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lks25\" (UniqueName: \"kubernetes.io/projected/8280243e-6c6c-4a19-8594-284b8f80e708-kube-api-access-lks25\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8280243e-6c6c-4a19-8594-284b8f80e708-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321927 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321927 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.321927 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.321857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.322095 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.322006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.322318 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.322117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8280243e-6c6c-4a19-8594-284b8f80e708-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.323880 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.323861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.324183 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.324163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8280243e-6c6c-4a19-8594-284b8f80e708-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.333630 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.333605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lks25\" (UniqueName: \"kubernetes.io/projected/8280243e-6c6c-4a19-8594-284b8f80e708-kube-api-access-lks25\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.441661 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.441629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:12.719440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:12.719415    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"]
Sep 16 19:33:12.721693 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:33:12.721660    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8280243e_6c6c_4a19_8594_284b8f80e708.slice/crio-c9d38979250db0feea86cecae0ac6681530735b9e83aed770b9708f257eba0c8 WatchSource:0}: Error finding container c9d38979250db0feea86cecae0ac6681530735b9e83aed770b9708f257eba0c8: Status 404 returned error can't find the container with id c9d38979250db0feea86cecae0ac6681530735b9e83aed770b9708f257eba0c8
Sep 16 19:33:13.668156 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:13.668086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" event={"ID":"8280243e-6c6c-4a19-8594-284b8f80e708","Type":"ContainerStarted","Data":"ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290"}
Sep 16 19:33:13.668668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:13.668169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" event={"ID":"8280243e-6c6c-4a19-8594-284b8f80e708","Type":"ContainerStarted","Data":"c9d38979250db0feea86cecae0ac6681530735b9e83aed770b9708f257eba0c8"}
Sep 16 19:33:15.680169 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:15.680137    2580 generic.go:356] "Generic (PLEG): container finished" podID="8280243e-6c6c-4a19-8594-284b8f80e708" containerID="ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290" exitCode=0
Sep 16 19:33:15.680545 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:15.680208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" event={"ID":"8280243e-6c6c-4a19-8594-284b8f80e708","Type":"ContainerDied","Data":"ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290"}
Sep 16 19:33:16.687069 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:16.687033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" event={"ID":"8280243e-6c6c-4a19-8594-284b8f80e708","Type":"ContainerStarted","Data":"95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a"}
Sep 16 19:33:16.711294 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:16.711245    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" podStartSLOduration=1.988611312 podStartE2EDuration="4.711227519s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.722616207s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:16.709181853 +0000 UTC m=+2976.970858876" watchObservedRunningTime="2026-09-16 19:33:16.711227519 +0000 UTC m=+2976.972904531"
Sep 16 19:33:17.442699 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:17.442665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:22.442761 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:22.442707    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:33:22.444269 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:22.444241    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:32.442470 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:32.442430    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:42.442914 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:42.442858    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:50.527153 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.527119    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"]
Sep 16 19:33:50.531022 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.530999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.534530 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.534506    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-zswn5\""
Sep 16 19:33:50.534823 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.534723    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 19:33:50.534953 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.534926    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:33:50.535205 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.535186    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 19:33:50.545510 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.545485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"]
Sep 16 19:33:50.565248 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.565215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.565248 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.565256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.565534 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.565289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.565534 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.565394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.565534 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.565449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.565534 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.565475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q62k\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-kube-api-access-9q62k\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.565534 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.565522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.565820 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.565569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.565820 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.565604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666037 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666216 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666216 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666216 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q62k\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-kube-api-access-9q62k\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666216 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666216 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666467 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666467 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666467 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666630 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666630 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666814 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.666920 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.666812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.667196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.667171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.668739 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.668714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.669027 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.669009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.685219 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.685189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.685447 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.685425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q62k\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-kube-api-access-9q62k\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-h7ss5\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:50.842813 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:50.842727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:51.120748 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:51.120713    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"]
Sep 16 19:33:51.121666 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:33:51.121635    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d66cb3_d7c7_4cd6_9ae6_8905606847e4.slice/crio-bc2590c839b9492dd484aa3d68467a11a7190343ac7b2cebc426165748969a1a WatchSource:0}: Error finding container bc2590c839b9492dd484aa3d68467a11a7190343ac7b2cebc426165748969a1a: Status 404 returned error can't find the container with id bc2590c839b9492dd484aa3d68467a11a7190343ac7b2cebc426165748969a1a
Sep 16 19:33:51.123769 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:51.123741    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:33:51.123843 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:51.123799    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:33:51.123843 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:51.123826    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:33:51.856664 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:51.856623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" event={"ID":"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4","Type":"ContainerStarted","Data":"57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5"}
Sep 16 19:33:51.857095 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:51.856685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" event={"ID":"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4","Type":"ContainerStarted","Data":"bc2590c839b9492dd484aa3d68467a11a7190343ac7b2cebc426165748969a1a"}
Sep 16 19:33:51.880463 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:51.880413    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" podStartSLOduration=1.880398126 podStartE2EDuration="1.880398126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:51.87746357 +0000 UTC m=+3012.139140623" watchObservedRunningTime="2026-09-16 19:33:51.880398126 +0000 UTC m=+3012.142075209"
Sep 16 19:33:52.442988 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:52.442933    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:52.843181 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:52.843100    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:52.848038 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:52.848014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:52.861931 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:52.861903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:33:52.862544 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:33:52.862526    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:34:02.442862 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:02.442822    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:34:12.442613 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:12.442570    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:34:22.442929 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:22.442866    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:34:32.463137 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:32.463086    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:34:32.475517 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:32.475487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:34:35.087493 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:35.087457    2580 generic.go:356] "Generic (PLEG): container finished" podID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerID="57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5" exitCode=0
Sep 16 19:34:35.087857 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:35.087516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" event={"ID":"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4","Type":"ContainerDied","Data":"57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5"}
Sep 16 19:34:35.087942 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:35.087928    2580 scope.go:117] "RemoveContainer" containerID="57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5"
Sep 16 19:34:35.088861 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:35.088836    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:34:35.088939 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:35.088925    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:34:35.088985 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:35.088977    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:34:36.095264 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:36.095228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" event={"ID":"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4","Type":"ContainerStarted","Data":"63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607"}
Sep 16 19:34:36.843477 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:36.843439    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:34:36.844515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:36.844486    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 19:34:37.843577 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:37.843528    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 19:34:37.863057 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:37.862921    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:34:38.843278 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:38.843234    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 19:34:39.843355 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:39.843318    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 19:34:40.844004 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:40.843959    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 19:34:41.847827 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:41.847795    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:34:41.848751 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:34:41.848731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:35:08.866519 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:08.866483    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"]
Sep 16 19:35:08.867120 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:08.866727    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy" containerID="cri-o://63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607" gracePeriod=30
Sep 16 19:35:11.850997 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:11.850952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:35:14.116529 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.116503    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:35:14.252998 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.252963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-envoy\") pod \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") "
Sep 16 19:35:14.253194 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-credential-socket\") pod \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") "
Sep 16 19:35:14.253194 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253075    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-certs\") pod \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") "
Sep 16 19:35:14.253324 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istiod-ca-cert\") pod \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") "
Sep 16 19:35:14.253324 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253263    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-token\") pod \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") "
Sep 16 19:35:14.253324 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-podinfo\") pod \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") "
Sep 16 19:35:14.253654 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-socket\") pod \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") "
Sep 16 19:35:14.253654 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253367    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q62k\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-kube-api-access-9q62k\") pod \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") "
Sep 16 19:35:14.253654 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-data\") pod \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\" (UID: \"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4\") "
Sep 16 19:35:14.253654 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253395    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" (UID: "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:14.253654 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" (UID: "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:14.253959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253706    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-credential-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:14.253959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253727    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:14.253959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253767    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" (UID: "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:14.253959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.253936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" (UID: "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:14.254204 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.254176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" (UID: "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:14.255818 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.255796    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" (UID: "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:14.255985 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.255965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" (UID: "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:14.255985 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.255976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" (UID: "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:35:14.256129 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.256074    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-kube-api-access-9q62k" (OuterVolumeSpecName: "kube-api-access-9q62k") pod "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" (UID: "e3d66cb3-d7c7-4cd6-9ae6-8905606847e4"). InnerVolumeSpecName "kube-api-access-9q62k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:14.303652 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.303626    2580 generic.go:356] "Generic (PLEG): container finished" podID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerID="63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607" exitCode=0
Sep 16 19:35:14.303777 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.303665    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"
Sep 16 19:35:14.303777 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.303681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" event={"ID":"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4","Type":"ContainerDied","Data":"63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607"}
Sep 16 19:35:14.303777 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.303728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5" event={"ID":"e3d66cb3-d7c7-4cd6-9ae6-8905606847e4","Type":"ContainerDied","Data":"bc2590c839b9492dd484aa3d68467a11a7190343ac7b2cebc426165748969a1a"}
Sep 16 19:35:14.303777 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.303749    2580 scope.go:117] "RemoveContainer" containerID="63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607"
Sep 16 19:35:14.313700 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.313678    2580 scope.go:117] "RemoveContainer" containerID="57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5"
Sep 16 19:35:14.322645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.322604    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"]
Sep 16 19:35:14.328461 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.328427    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-h7ss5"]
Sep 16 19:35:14.328748 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.328731    2580 scope.go:117] "RemoveContainer" containerID="63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607"
Sep 16 19:35:14.329182 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:35:14.329163    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607\": container with ID starting with 63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607 not found: ID does not exist" containerID="63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607"
Sep 16 19:35:14.329268 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.329191    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607"} err="failed to get container status \"63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607\": rpc error: code = NotFound desc = could not find container \"63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607\": container with ID starting with 63a6b27997487af99819e10b674defe12a83a0e11c156972dfcef8a9708ff607 not found: ID does not exist"
Sep 16 19:35:14.329268 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.329209    2580 scope.go:117] "RemoveContainer" containerID="57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5"
Sep 16 19:35:14.329551 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:35:14.329530    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5\": container with ID starting with 57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5 not found: ID does not exist" containerID="57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5"
Sep 16 19:35:14.329596 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.329557    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5"} err="failed to get container status \"57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5\": rpc error: code = NotFound desc = could not find container \"57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5\": container with ID starting with 57d0a55359d67c481fdfa536d9e0a8469d7eb184d65c11e2890cc299162ec7d5 not found: ID does not exist"
Sep 16 19:35:14.354569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.354545    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-workload-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:14.354569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.354564    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q62k\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-kube-api-access-9q62k\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:14.354703 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.354575    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-data\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:14.354703 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.354585    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-envoy\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:14.354703 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.354599    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istiod-ca-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:14.354703 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.354609    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-token\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:14.354703 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:14.354616    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4-istio-podinfo\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:16.276551 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:16.276518    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" path="/var/lib/kubelet/pods/e3d66cb3-d7c7-4cd6-9ae6-8905606847e4/volumes"
Sep 16 19:35:22.549353 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:22.549317    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"]
Sep 16 19:35:37.572050 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:37.571987    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main" containerID="cri-o://95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a" gracePeriod=45
Sep 16 19:35:38.224569 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.224544    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:35:38.352595 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.352521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-kserve-provision-location\") pod \"8280243e-6c6c-4a19-8594-284b8f80e708\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") "
Sep 16 19:35:38.352595 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.352581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-tmp-dir\") pod \"8280243e-6c6c-4a19-8594-284b8f80e708\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") "
Sep 16 19:35:38.352807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.352601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8280243e-6c6c-4a19-8594-284b8f80e708-cabundle-cert\") pod \"8280243e-6c6c-4a19-8594-284b8f80e708\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") "
Sep 16 19:35:38.352807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.352629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-dshm\") pod \"8280243e-6c6c-4a19-8594-284b8f80e708\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") "
Sep 16 19:35:38.352807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.352660    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lks25\" (UniqueName: \"kubernetes.io/projected/8280243e-6c6c-4a19-8594-284b8f80e708-kube-api-access-lks25\") pod \"8280243e-6c6c-4a19-8594-284b8f80e708\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") "
Sep 16 19:35:38.352807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.352676    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8280243e-6c6c-4a19-8594-284b8f80e708-tls-certs\") pod \"8280243e-6c6c-4a19-8594-284b8f80e708\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") "
Sep 16 19:35:38.352807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.352707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-home\") pod \"8280243e-6c6c-4a19-8594-284b8f80e708\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") "
Sep 16 19:35:38.352807 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.352755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-model-cache\") pod \"8280243e-6c6c-4a19-8594-284b8f80e708\" (UID: \"8280243e-6c6c-4a19-8594-284b8f80e708\") "
Sep 16 19:35:38.353156 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.353019    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8280243e-6c6c-4a19-8594-284b8f80e708-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8280243e-6c6c-4a19-8594-284b8f80e708" (UID: "8280243e-6c6c-4a19-8594-284b8f80e708"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:38.353217 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.353185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-model-cache" (OuterVolumeSpecName: "model-cache") pod "8280243e-6c6c-4a19-8594-284b8f80e708" (UID: "8280243e-6c6c-4a19-8594-284b8f80e708"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:38.353509 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.353476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-home" (OuterVolumeSpecName: "home") pod "8280243e-6c6c-4a19-8594-284b8f80e708" (UID: "8280243e-6c6c-4a19-8594-284b8f80e708"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:38.355127 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.355105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-dshm" (OuterVolumeSpecName: "dshm") pod "8280243e-6c6c-4a19-8594-284b8f80e708" (UID: "8280243e-6c6c-4a19-8594-284b8f80e708"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:38.355352 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.355323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8280243e-6c6c-4a19-8594-284b8f80e708-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8280243e-6c6c-4a19-8594-284b8f80e708" (UID: "8280243e-6c6c-4a19-8594-284b8f80e708"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:38.355429 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.355334    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8280243e-6c6c-4a19-8594-284b8f80e708-kube-api-access-lks25" (OuterVolumeSpecName: "kube-api-access-lks25") pod "8280243e-6c6c-4a19-8594-284b8f80e708" (UID: "8280243e-6c6c-4a19-8594-284b8f80e708"). InnerVolumeSpecName "kube-api-access-lks25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:38.366460 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.366434    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8280243e-6c6c-4a19-8594-284b8f80e708" (UID: "8280243e-6c6c-4a19-8594-284b8f80e708"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:38.376413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.376387    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8280243e-6c6c-4a19-8594-284b8f80e708" (UID: "8280243e-6c6c-4a19-8594-284b8f80e708"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:38.422161 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.422139    2580 generic.go:356] "Generic (PLEG): container finished" podID="8280243e-6c6c-4a19-8594-284b8f80e708" containerID="95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a" exitCode=0
Sep 16 19:35:38.422254 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.422202    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"
Sep 16 19:35:38.422298 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.422203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" event={"ID":"8280243e-6c6c-4a19-8594-284b8f80e708","Type":"ContainerDied","Data":"95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a"}
Sep 16 19:35:38.422337 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.422315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5" event={"ID":"8280243e-6c6c-4a19-8594-284b8f80e708","Type":"ContainerDied","Data":"c9d38979250db0feea86cecae0ac6681530735b9e83aed770b9708f257eba0c8"}
Sep 16 19:35:38.422372 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.422336    2580 scope.go:117] "RemoveContainer" containerID="95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a"
Sep 16 19:35:38.431511 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.431488    2580 scope.go:117] "RemoveContainer" containerID="ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290"
Sep 16 19:35:38.439260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.439241    2580 scope.go:117] "RemoveContainer" containerID="95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a"
Sep 16 19:35:38.439514 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:35:38.439498    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a\": container with ID starting with 95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a not found: ID does not exist" containerID="95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a"
Sep 16 19:35:38.439560 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.439522    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a"} err="failed to get container status \"95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a\": rpc error: code = NotFound desc = could not find container \"95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a\": container with ID starting with 95ebcdc7ccf454adaa740766e661079bc8b3d8365a6a6bfdfcee0e6017ccb27a not found: ID does not exist"
Sep 16 19:35:38.439560 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.439537    2580 scope.go:117] "RemoveContainer" containerID="ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290"
Sep 16 19:35:38.439788 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:35:38.439772    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290\": container with ID starting with ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290 not found: ID does not exist" containerID="ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290"
Sep 16 19:35:38.439827 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.439794    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290"} err="failed to get container status \"ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290\": rpc error: code = NotFound desc = could not find container \"ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290\": container with ID starting with ba07de9607457282c429e0ac25a8554eb9d0a16cf81e04a817e308118b994290 not found: ID does not exist"
Sep 16 19:35:38.445906 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.445870    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"]
Sep 16 19:35:38.450538 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.450518    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-749d6db49-6grx5"]
Sep 16 19:35:38.454151 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.454132    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:38.454214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.454153    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8280243e-6c6c-4a19-8594-284b8f80e708-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:38.454214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.454162    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:38.454214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.454170    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lks25\" (UniqueName: \"kubernetes.io/projected/8280243e-6c6c-4a19-8594-284b8f80e708-kube-api-access-lks25\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:38.454214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.454179    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8280243e-6c6c-4a19-8594-284b8f80e708-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:38.454214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.454188    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:38.454214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.454196    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:38.454214 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:38.454203    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8280243e-6c6c-4a19-8594-284b8f80e708-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:35:40.277119 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:35:40.277086    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" path="/var/lib/kubelet/pods/8280243e-6c6c-4a19-8594-284b8f80e708/volumes"
Sep 16 19:36:14.719388 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719295    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"]
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719712    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719731    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719749    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719758    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719797    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="storage-initializer"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719807    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="storage-initializer"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719818    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719827    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719932    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719948    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8280243e-6c6c-4a19-8594-284b8f80e708" containerName="main"
Sep 16 19:36:14.719987 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.719959    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d66cb3-d7c7-4cd6-9ae6-8905606847e4" containerName="istio-proxy"
Sep 16 19:36:14.722370 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.722348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.725091 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.725057    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 19:36:14.725238 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.725058    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 19:36:14.725238 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.725116    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 19:36:14.725432 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.725413    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:36:14.725490 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.725437    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 19:36:14.725665 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.725648    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-gldcd\""
Sep 16 19:36:14.739202 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.739182    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"]
Sep 16 19:36:14.754779 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.754755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.754885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.754797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.754885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.754814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c001085-fac5-4d5a-8f10-4701e2d2b21d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.754885 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.754836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.755018 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.754901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.755018 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.754939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.755018 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.754969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.755018 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.754996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g7q5\" (UniqueName: \"kubernetes.io/projected/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kube-api-access-6g7q5\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.855580 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.855553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.855708 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.855591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g7q5\" (UniqueName: \"kubernetes.io/projected/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kube-api-access-6g7q5\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.855757 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.855712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.855817 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.855775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.855817 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.855797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c001085-fac5-4d5a-8f10-4701e2d2b21d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.855961 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.855828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.855961 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.855852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.855961 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.855871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.855961 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.855883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.856170 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.856127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.856219 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.856192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.856289 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.856267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.856527 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.856508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c001085-fac5-4d5a-8f10-4701e2d2b21d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.858238 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.858213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.858480 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.858465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:14.864549 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:14.864531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g7q5\" (UniqueName: \"kubernetes.io/projected/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kube-api-access-6g7q5\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:15.033266 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:15.033180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:15.312750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:15.312721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"]
Sep 16 19:36:15.314109 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:36:15.314081    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c001085_fac5_4d5a_8f10_4701e2d2b21d.slice/crio-95df2367a8d525a2e365be3a5452902067c70caeef54d6087a69fb3a644a43d9 WatchSource:0}: Error finding container 95df2367a8d525a2e365be3a5452902067c70caeef54d6087a69fb3a644a43d9: Status 404 returned error can't find the container with id 95df2367a8d525a2e365be3a5452902067c70caeef54d6087a69fb3a644a43d9
Sep 16 19:36:15.589917 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:15.589818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" event={"ID":"0c001085-fac5-4d5a-8f10-4701e2d2b21d","Type":"ContainerStarted","Data":"d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd"}
Sep 16 19:36:15.589917 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:15.589882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" event={"ID":"0c001085-fac5-4d5a-8f10-4701e2d2b21d","Type":"ContainerStarted","Data":"95df2367a8d525a2e365be3a5452902067c70caeef54d6087a69fb3a644a43d9"}
Sep 16 19:36:18.607104 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:18.607068    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerID="d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd" exitCode=0
Sep 16 19:36:18.607460 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:18.607130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" event={"ID":"0c001085-fac5-4d5a-8f10-4701e2d2b21d","Type":"ContainerDied","Data":"d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd"}
Sep 16 19:36:19.614655 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:19.614622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" event={"ID":"0c001085-fac5-4d5a-8f10-4701e2d2b21d","Type":"ContainerStarted","Data":"07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512"}
Sep 16 19:36:19.637321 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:19.637269    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podStartSLOduration=2.932054731 podStartE2EDuration="5.637241587s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.705186856s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:19.635581918 +0000 UTC m=+3159.897258930" watchObservedRunningTime="2026-09-16 19:36:19.637241587 +0000 UTC m=+3159.898918598"
Sep 16 19:36:20.033749 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:20.033715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:25.033585 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:25.033551    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:36:25.034971 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:25.034943    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:36:35.033851 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:35.033808    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:36:45.033797 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:45.033754    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:36:55.034196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:36:55.034158    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:05.034289 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:05.034239    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:15.033833 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:15.033794    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:25.033796 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:25.033754    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:35.043621 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:35.043586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:37:35.051176 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:35.051155    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:37:36.714245 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:36.714208    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"]
Sep 16 19:37:52.026575 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.026523    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main" containerID="cri-o://07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512" gracePeriod=45
Sep 16 19:37:52.044409 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.044375    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:52.564976 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.564954    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:37:52.610029 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610004    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-model-cache\") pod \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") "
Sep 16 19:37:52.610163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610033    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g7q5\" (UniqueName: \"kubernetes.io/projected/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kube-api-access-6g7q5\") pod \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") "
Sep 16 19:37:52.610163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c001085-fac5-4d5a-8f10-4701e2d2b21d-cabundle-cert\") pod \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") "
Sep 16 19:37:52.610163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tmp-dir\") pod \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") "
Sep 16 19:37:52.610163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610123    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-home\") pod \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") "
Sep 16 19:37:52.610163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-dshm\") pod \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") "
Sep 16 19:37:52.610426 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kserve-provision-location\") pod \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") "
Sep 16 19:37:52.610426 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tls-certs\") pod \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\" (UID: \"0c001085-fac5-4d5a-8f10-4701e2d2b21d\") "
Sep 16 19:37:52.610426 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610299    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c001085-fac5-4d5a-8f10-4701e2d2b21d" (UID: "0c001085-fac5-4d5a-8f10-4701e2d2b21d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:52.610684 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610518    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c001085-fac5-4d5a-8f10-4701e2d2b21d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c001085-fac5-4d5a-8f10-4701e2d2b21d" (UID: "0c001085-fac5-4d5a-8f10-4701e2d2b21d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:52.610684 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c001085-fac5-4d5a-8f10-4701e2d2b21d" (UID: "0c001085-fac5-4d5a-8f10-4701e2d2b21d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:52.610843 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610813    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-home" (OuterVolumeSpecName: "home") pod "0c001085-fac5-4d5a-8f10-4701e2d2b21d" (UID: "0c001085-fac5-4d5a-8f10-4701e2d2b21d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:52.610988 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610961    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:37:52.610988 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610983    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:37:52.611130 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.610996    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:37:52.611130 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.611011    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c001085-fac5-4d5a-8f10-4701e2d2b21d-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:37:52.612518 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.612494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kube-api-access-6g7q5" (OuterVolumeSpecName: "kube-api-access-6g7q5") pod "0c001085-fac5-4d5a-8f10-4701e2d2b21d" (UID: "0c001085-fac5-4d5a-8f10-4701e2d2b21d"). InnerVolumeSpecName "kube-api-access-6g7q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:52.613004 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.612981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-dshm" (OuterVolumeSpecName: "dshm") pod "0c001085-fac5-4d5a-8f10-4701e2d2b21d" (UID: "0c001085-fac5-4d5a-8f10-4701e2d2b21d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:52.613113 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.612981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c001085-fac5-4d5a-8f10-4701e2d2b21d" (UID: "0c001085-fac5-4d5a-8f10-4701e2d2b21d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:52.633946 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.633921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c001085-fac5-4d5a-8f10-4701e2d2b21d" (UID: "0c001085-fac5-4d5a-8f10-4701e2d2b21d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:52.711974 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.711951    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:37:52.711974 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.711976    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c001085-fac5-4d5a-8f10-4701e2d2b21d-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:37:52.712127 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.711987    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g7q5\" (UniqueName: \"kubernetes.io/projected/0c001085-fac5-4d5a-8f10-4701e2d2b21d-kube-api-access-6g7q5\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:37:52.712127 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:52.711998    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c001085-fac5-4d5a-8f10-4701e2d2b21d-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:37:53.078955 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.078867    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerID="07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512" exitCode=0
Sep 16 19:37:53.078955 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.078937    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"
Sep 16 19:37:53.079444 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.078937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" event={"ID":"0c001085-fac5-4d5a-8f10-4701e2d2b21d","Type":"ContainerDied","Data":"07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512"}
Sep 16 19:37:53.079444 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.079066    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n" event={"ID":"0c001085-fac5-4d5a-8f10-4701e2d2b21d","Type":"ContainerDied","Data":"95df2367a8d525a2e365be3a5452902067c70caeef54d6087a69fb3a644a43d9"}
Sep 16 19:37:53.079444 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.079103    2580 scope.go:117] "RemoveContainer" containerID="07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512"
Sep 16 19:37:53.088183 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.088165    2580 scope.go:117] "RemoveContainer" containerID="d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd"
Sep 16 19:37:53.095927 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.095910    2580 scope.go:117] "RemoveContainer" containerID="07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512"
Sep 16 19:37:53.096190 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:37:53.096170    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512\": container with ID starting with 07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512 not found: ID does not exist" containerID="07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512"
Sep 16 19:37:53.096239 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.096201    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512"} err="failed to get container status \"07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512\": rpc error: code = NotFound desc = could not find container \"07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512\": container with ID starting with 07b69ba12f41a234ef04d6228ce0aa90a2cb6734b4df093ce6b8bd2a7281e512 not found: ID does not exist"
Sep 16 19:37:53.096239 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.096217    2580 scope.go:117] "RemoveContainer" containerID="d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd"
Sep 16 19:37:53.096482 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:37:53.096464    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd\": container with ID starting with d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd not found: ID does not exist" containerID="d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd"
Sep 16 19:37:53.096529 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.096488    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd"} err="failed to get container status \"d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd\": rpc error: code = NotFound desc = could not find container \"d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd\": container with ID starting with d42f1a0b7833f2ec4ea0fb0f15dcdb9947bd89693bed17603bc5cc2929158efd not found: ID does not exist"
Sep 16 19:37:53.102015 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.101996    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"]
Sep 16 19:37:53.107848 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:53.107825    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-98f98877d-m6p7n"]
Sep 16 19:37:54.275417 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:37:54.275387    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" path="/var/lib/kubelet/pods/0c001085-fac5-4d5a-8f10-4701e2d2b21d/volumes"
Sep 16 19:38:35.144149 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.144112    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"]
Sep 16 19:38:35.144550 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.144458    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="storage-initializer"
Sep 16 19:38:35.144550 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.144470    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="storage-initializer"
Sep 16 19:38:35.144550 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.144479    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main"
Sep 16 19:38:35.144550 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.144484    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main"
Sep 16 19:38:35.144721 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.144573    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c001085-fac5-4d5a-8f10-4701e2d2b21d" containerName="main"
Sep 16 19:38:35.147421 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.147400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:38:35.149640 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.149620    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:38:35.150442 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.150420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-bx87j\""
Sep 16 19:38:35.150442 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.150438    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:38:35.150596 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.150420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-w85c5\""
Sep 16 19:38:35.150596 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.150558    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:38:35.160820 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.160798    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"]
Sep 16 19:38:35.244484 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.244454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea848b14-0bc7-427d-8c14-5e1e68895607-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b\" (UID: \"ea848b14-0bc7-427d-8c14-5e1e68895607\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:38:35.244616 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.244514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrghp\" (UniqueName: \"kubernetes.io/projected/ea848b14-0bc7-427d-8c14-5e1e68895607-kube-api-access-mrghp\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b\" (UID: \"ea848b14-0bc7-427d-8c14-5e1e68895607\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:38:35.345617 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.345592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea848b14-0bc7-427d-8c14-5e1e68895607-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b\" (UID: \"ea848b14-0bc7-427d-8c14-5e1e68895607\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:38:35.345748 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.345665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrghp\" (UniqueName: \"kubernetes.io/projected/ea848b14-0bc7-427d-8c14-5e1e68895607-kube-api-access-mrghp\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b\" (UID: \"ea848b14-0bc7-427d-8c14-5e1e68895607\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:38:35.348328 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.348288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea848b14-0bc7-427d-8c14-5e1e68895607-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b\" (UID: \"ea848b14-0bc7-427d-8c14-5e1e68895607\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:38:35.354440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.354415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrghp\" (UniqueName: \"kubernetes.io/projected/ea848b14-0bc7-427d-8c14-5e1e68895607-kube-api-access-mrghp\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b\" (UID: \"ea848b14-0bc7-427d-8c14-5e1e68895607\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:38:35.457979 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.457951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:38:35.755805 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:35.755723    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"]
Sep 16 19:38:35.758931 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:38:35.758884    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea848b14_0bc7_427d_8c14_5e1e68895607.slice/crio-7c8d66d935a5ca8a5a132198fdd4c90ddffe9368c969d03a88d2f084b52aaf6e WatchSource:0}: Error finding container 7c8d66d935a5ca8a5a132198fdd4c90ddffe9368c969d03a88d2f084b52aaf6e: Status 404 returned error can't find the container with id 7c8d66d935a5ca8a5a132198fdd4c90ddffe9368c969d03a88d2f084b52aaf6e
Sep 16 19:38:36.276678 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:36.276644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" event={"ID":"ea848b14-0bc7-427d-8c14-5e1e68895607","Type":"ContainerStarted","Data":"13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79"}
Sep 16 19:38:36.277054 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:36.276693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" event={"ID":"ea848b14-0bc7-427d-8c14-5e1e68895607","Type":"ContainerStarted","Data":"7c8d66d935a5ca8a5a132198fdd4c90ddffe9368c969d03a88d2f084b52aaf6e"}
Sep 16 19:38:36.277054 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:36.276778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:38:36.313319 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:38:36.313270    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" podStartSLOduration=1.313254809 podStartE2EDuration="1.313254809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:36.311412932 +0000 UTC m=+3296.573089944" watchObservedRunningTime="2026-09-16 19:38:36.313254809 +0000 UTC m=+3296.574931829"
Sep 16 19:39:07.282948 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:39:07.282841    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:41:20.522977 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.522948    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"]
Sep 16 19:41:20.526451 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.526433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.529507 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.529481    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:41:20.529507 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.529500    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:41:20.529698 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.529609    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-9766v\""
Sep 16 19:41:20.529698 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.529665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:41:20.529809 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.529748    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:41:20.529852 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.529815    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:41:20.539661 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.539636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"]
Sep 16 19:41:20.544747 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.544725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqg6b\" (UniqueName: \"kubernetes.io/projected/cfba2c2e-4d57-4e22-8d75-1154202724b9-kube-api-access-fqg6b\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.544852 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.544760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-tmp-dir\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.544852 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.544791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-kserve-provision-location\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.544959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.544856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfba2c2e-4d57-4e22-8d75-1154202724b9-tls-certs\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.544959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.544873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-dshm\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.544959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.544923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-model-cache\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.544959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.544941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-home\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.545124 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.544969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfba2c2e-4d57-4e22-8d75-1154202724b9-cabundle-cert\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.645751 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.645723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-model-cache\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.645879 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.645761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-home\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.645879 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.645784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfba2c2e-4d57-4e22-8d75-1154202724b9-cabundle-cert\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646042 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.645932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqg6b\" (UniqueName: \"kubernetes.io/projected/cfba2c2e-4d57-4e22-8d75-1154202724b9-kube-api-access-fqg6b\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646042 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.645968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-tmp-dir\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646042 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.645992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-kserve-provision-location\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646208 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.646051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfba2c2e-4d57-4e22-8d75-1154202724b9-tls-certs\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646208 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.646068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-dshm\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646208 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.646136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-model-cache\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646208 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.646180    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-home\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646418 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.646265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-tmp-dir\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646475 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.646445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-kserve-provision-location\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.646576 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.646556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfba2c2e-4d57-4e22-8d75-1154202724b9-cabundle-cert\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.648432 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.648412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-dshm\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.648658 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.648642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfba2c2e-4d57-4e22-8d75-1154202724b9-tls-certs\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.653745 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.653728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqg6b\" (UniqueName: \"kubernetes.io/projected/cfba2c2e-4d57-4e22-8d75-1154202724b9-kube-api-access-fqg6b\") pod \"stop-feature-test-kserve-7b545f4984-ng6gs\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:20.837158 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:20.837078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:21.116119 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:21.115968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"]
Sep 16 19:41:21.118637 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:41:21.118609    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfba2c2e_4d57_4e22_8d75_1154202724b9.slice/crio-1ead6a270056a65f0da23f6638af2814907d7e12efc5a192be7fb40dd915f055 WatchSource:0}: Error finding container 1ead6a270056a65f0da23f6638af2814907d7e12efc5a192be7fb40dd915f055: Status 404 returned error can't find the container with id 1ead6a270056a65f0da23f6638af2814907d7e12efc5a192be7fb40dd915f055
Sep 16 19:41:22.005502 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:22.005467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" event={"ID":"cfba2c2e-4d57-4e22-8d75-1154202724b9","Type":"ContainerStarted","Data":"1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424"}
Sep 16 19:41:22.005864 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:22.005520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" event={"ID":"cfba2c2e-4d57-4e22-8d75-1154202724b9","Type":"ContainerStarted","Data":"1ead6a270056a65f0da23f6638af2814907d7e12efc5a192be7fb40dd915f055"}
Sep 16 19:41:24.017632 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:24.017596    2580 generic.go:356] "Generic (PLEG): container finished" podID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerID="1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424" exitCode=0
Sep 16 19:41:24.018123 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:24.017677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" event={"ID":"cfba2c2e-4d57-4e22-8d75-1154202724b9","Type":"ContainerDied","Data":"1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424"}
Sep 16 19:41:25.025288 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:25.025250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" event={"ID":"cfba2c2e-4d57-4e22-8d75-1154202724b9","Type":"ContainerStarted","Data":"f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431"}
Sep 16 19:41:25.049870 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:25.049822    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podStartSLOduration=2.96233013 podStartE2EDuration="5.049807768s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.087477638s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:25.047146962 +0000 UTC m=+3465.308823972" watchObservedRunningTime="2026-09-16 19:41:25.049807768 +0000 UTC m=+3465.311484778"
Sep 16 19:41:25.837473 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:25.837441    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:30.837880 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:30.837842    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:41:30.839397 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:30.839362    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:41:40.838380 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:40.838335    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:41:50.775400 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:50.775360    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"]
Sep 16 19:41:50.838263 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:50.838218    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:41:57.282764 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:41:57.282726    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" podUID="ea848b14-0bc7-427d-8c14-5e1e68895607" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:00.837728 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:00.837635    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:42:05.803197 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:05.803150    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" podUID="ea848b14-0bc7-427d-8c14-5e1e68895607" containerName="main" containerID="cri-o://13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79" gracePeriod=45
Sep 16 19:42:07.282523 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:07.282477    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" podUID="ea848b14-0bc7-427d-8c14-5e1e68895607" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:10.837883 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:10.837832    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:42:17.283033 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:17.282994    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" podUID="ea848b14-0bc7-427d-8c14-5e1e68895607" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:17.283472 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:17.283097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:42:20.837726 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:20.837679    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:42:27.283286 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:27.283245    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" podUID="ea848b14-0bc7-427d-8c14-5e1e68895607" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:30.837685 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:30.837639    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:42:36.041948 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.041923    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:42:36.119189 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.119115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea848b14-0bc7-427d-8c14-5e1e68895607-tls-certs\") pod \"ea848b14-0bc7-427d-8c14-5e1e68895607\" (UID: \"ea848b14-0bc7-427d-8c14-5e1e68895607\") "
Sep 16 19:42:36.119313 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.119243    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrghp\" (UniqueName: \"kubernetes.io/projected/ea848b14-0bc7-427d-8c14-5e1e68895607-kube-api-access-mrghp\") pod \"ea848b14-0bc7-427d-8c14-5e1e68895607\" (UID: \"ea848b14-0bc7-427d-8c14-5e1e68895607\") "
Sep 16 19:42:36.121331 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.121298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea848b14-0bc7-427d-8c14-5e1e68895607-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea848b14-0bc7-427d-8c14-5e1e68895607" (UID: "ea848b14-0bc7-427d-8c14-5e1e68895607"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:36.121436 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.121392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea848b14-0bc7-427d-8c14-5e1e68895607-kube-api-access-mrghp" (OuterVolumeSpecName: "kube-api-access-mrghp") pod "ea848b14-0bc7-427d-8c14-5e1e68895607" (UID: "ea848b14-0bc7-427d-8c14-5e1e68895607"). InnerVolumeSpecName "kube-api-access-mrghp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:36.220689 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.220662    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea848b14-0bc7-427d-8c14-5e1e68895607-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:42:36.220689 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.220687    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrghp\" (UniqueName: \"kubernetes.io/projected/ea848b14-0bc7-427d-8c14-5e1e68895607-kube-api-access-mrghp\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:42:36.392645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.392550    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea848b14-0bc7-427d-8c14-5e1e68895607" containerID="13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79" exitCode=0
Sep 16 19:42:36.392645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.392603    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"
Sep 16 19:42:36.392645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.392637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" event={"ID":"ea848b14-0bc7-427d-8c14-5e1e68895607","Type":"ContainerDied","Data":"13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79"}
Sep 16 19:42:36.392876 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.392708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b" event={"ID":"ea848b14-0bc7-427d-8c14-5e1e68895607","Type":"ContainerDied","Data":"7c8d66d935a5ca8a5a132198fdd4c90ddffe9368c969d03a88d2f084b52aaf6e"}
Sep 16 19:42:36.392876 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.392732    2580 scope.go:117] "RemoveContainer" containerID="13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79"
Sep 16 19:42:36.401280 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.401257    2580 scope.go:117] "RemoveContainer" containerID="13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79"
Sep 16 19:42:36.401570 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:42:36.401551    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79\": container with ID starting with 13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79 not found: ID does not exist" containerID="13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79"
Sep 16 19:42:36.401636 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.401579    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79"} err="failed to get container status \"13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79\": rpc error: code = NotFound desc = could not find container \"13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79\": container with ID starting with 13f5dc1667cd0c2e0e775bd0e69e5ecd3f00c115a5b3bb65e17702a6d1a7cc79 not found: ID does not exist"
Sep 16 19:42:36.407863 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.407841    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"]
Sep 16 19:42:36.412474 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:36.412453    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schel6b8b"]
Sep 16 19:42:38.276985 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:38.276951    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea848b14-0bc7-427d-8c14-5e1e68895607" path="/var/lib/kubelet/pods/ea848b14-0bc7-427d-8c14-5e1e68895607/volumes"
Sep 16 19:42:40.838405 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:40.838360    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:42:47.135442 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.135414    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"]
Sep 16 19:42:47.135802 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.135788    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea848b14-0bc7-427d-8c14-5e1e68895607" containerName="main"
Sep 16 19:42:47.135802 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.135801    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea848b14-0bc7-427d-8c14-5e1e68895607" containerName="main"
Sep 16 19:42:47.135879 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.135865    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea848b14-0bc7-427d-8c14-5e1e68895607" containerName="main"
Sep 16 19:42:47.140286 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.140266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:42:47.145665 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.145644    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 19:42:47.145784 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.145678    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 19:42:47.145784 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.145720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-djbgh\""
Sep 16 19:42:47.145784 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.145644    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 19:42:47.146048 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.145814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-btkfj\""
Sep 16 19:42:47.156050 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.156030    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"]
Sep 16 19:42:47.317949 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.317912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7dt\" (UniqueName: \"kubernetes.io/projected/28b3aa3c-6c8e-4c13-822f-ea770478f845-kube-api-access-fg7dt\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv\" (UID: \"28b3aa3c-6c8e-4c13-822f-ea770478f845\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:42:47.318131 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.317978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b3aa3c-6c8e-4c13-822f-ea770478f845-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv\" (UID: \"28b3aa3c-6c8e-4c13-822f-ea770478f845\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:42:47.418770 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.418691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7dt\" (UniqueName: \"kubernetes.io/projected/28b3aa3c-6c8e-4c13-822f-ea770478f845-kube-api-access-fg7dt\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv\" (UID: \"28b3aa3c-6c8e-4c13-822f-ea770478f845\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:42:47.418770 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.418743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b3aa3c-6c8e-4c13-822f-ea770478f845-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv\" (UID: \"28b3aa3c-6c8e-4c13-822f-ea770478f845\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:42:47.421613 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.421577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b3aa3c-6c8e-4c13-822f-ea770478f845-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv\" (UID: \"28b3aa3c-6c8e-4c13-822f-ea770478f845\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:42:47.429019 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.428998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7dt\" (UniqueName: \"kubernetes.io/projected/28b3aa3c-6c8e-4c13-822f-ea770478f845-kube-api-access-fg7dt\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv\" (UID: \"28b3aa3c-6c8e-4c13-822f-ea770478f845\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:42:47.449822 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.449795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:42:47.724762 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:47.724668    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"]
Sep 16 19:42:47.728455 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:42:47.728429    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b3aa3c_6c8e_4c13_822f_ea770478f845.slice/crio-30ab78a649032f0dab5c6893ca5ed04ba0d0500f85f1d6233d858112197a6ce1 WatchSource:0}: Error finding container 30ab78a649032f0dab5c6893ca5ed04ba0d0500f85f1d6233d858112197a6ce1: Status 404 returned error can't find the container with id 30ab78a649032f0dab5c6893ca5ed04ba0d0500f85f1d6233d858112197a6ce1
Sep 16 19:42:48.462196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:48.462164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" event={"ID":"28b3aa3c-6c8e-4c13-822f-ea770478f845","Type":"ContainerStarted","Data":"9a3df2c3e799ef8626ed8797be9d7d2a5f09997bedc173c3ab6607ffe5332a6d"}
Sep 16 19:42:48.462633 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:48.462239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" event={"ID":"28b3aa3c-6c8e-4c13-822f-ea770478f845","Type":"ContainerStarted","Data":"30ab78a649032f0dab5c6893ca5ed04ba0d0500f85f1d6233d858112197a6ce1"}
Sep 16 19:42:48.462633 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:48.462281    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:42:48.484185 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:48.484141    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" podStartSLOduration=1.484126942 podStartE2EDuration="1.484126942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:48.482313493 +0000 UTC m=+3548.743990542" watchObservedRunningTime="2026-09-16 19:42:48.484126942 +0000 UTC m=+3548.745803951"
Sep 16 19:42:50.837931 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:42:50.837865    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:43:00.838012 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:00.837960    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:43:10.847656 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:10.847625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:43:10.855036 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:10.855016    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:43:12.691681 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:12.691650    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"]
Sep 16 19:43:19.469124 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:19.469094    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:43:27.714123 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:27.714067    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" containerID="cri-o://f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431" gracePeriod=45
Sep 16 19:43:27.848406 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:27.848350    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:43:28.446318 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.446294    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:43:28.557735 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.557667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-home\") pod \"cfba2c2e-4d57-4e22-8d75-1154202724b9\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") "
Sep 16 19:43:28.557877 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.557765    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-dshm\") pod \"cfba2c2e-4d57-4e22-8d75-1154202724b9\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") "
Sep 16 19:43:28.557877 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.557791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfba2c2e-4d57-4e22-8d75-1154202724b9-cabundle-cert\") pod \"cfba2c2e-4d57-4e22-8d75-1154202724b9\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") "
Sep 16 19:43:28.557877 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.557827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfba2c2e-4d57-4e22-8d75-1154202724b9-tls-certs\") pod \"cfba2c2e-4d57-4e22-8d75-1154202724b9\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") "
Sep 16 19:43:28.557877 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.557873    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-model-cache\") pod \"cfba2c2e-4d57-4e22-8d75-1154202724b9\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") "
Sep 16 19:43:28.558131 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.557916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqg6b\" (UniqueName: \"kubernetes.io/projected/cfba2c2e-4d57-4e22-8d75-1154202724b9-kube-api-access-fqg6b\") pod \"cfba2c2e-4d57-4e22-8d75-1154202724b9\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") "
Sep 16 19:43:28.558131 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.558031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-kserve-provision-location\") pod \"cfba2c2e-4d57-4e22-8d75-1154202724b9\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") "
Sep 16 19:43:28.558131 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.558081    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-tmp-dir\") pod \"cfba2c2e-4d57-4e22-8d75-1154202724b9\" (UID: \"cfba2c2e-4d57-4e22-8d75-1154202724b9\") "
Sep 16 19:43:28.558286 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.558155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfba2c2e-4d57-4e22-8d75-1154202724b9" (UID: "cfba2c2e-4d57-4e22-8d75-1154202724b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:28.558286 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.558241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfba2c2e-4d57-4e22-8d75-1154202724b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfba2c2e-4d57-4e22-8d75-1154202724b9" (UID: "cfba2c2e-4d57-4e22-8d75-1154202724b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:28.558397 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.558385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-home" (OuterVolumeSpecName: "home") pod "cfba2c2e-4d57-4e22-8d75-1154202724b9" (UID: "cfba2c2e-4d57-4e22-8d75-1154202724b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:28.558455 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.558407    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:43:28.558455 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.558426    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfba2c2e-4d57-4e22-8d75-1154202724b9-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:43:28.560224 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.560201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfba2c2e-4d57-4e22-8d75-1154202724b9-kube-api-access-fqg6b" (OuterVolumeSpecName: "kube-api-access-fqg6b") pod "cfba2c2e-4d57-4e22-8d75-1154202724b9" (UID: "cfba2c2e-4d57-4e22-8d75-1154202724b9"). InnerVolumeSpecName "kube-api-access-fqg6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:28.560427 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.560407    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfba2c2e-4d57-4e22-8d75-1154202724b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfba2c2e-4d57-4e22-8d75-1154202724b9" (UID: "cfba2c2e-4d57-4e22-8d75-1154202724b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:28.560507 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.560473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-dshm" (OuterVolumeSpecName: "dshm") pod "cfba2c2e-4d57-4e22-8d75-1154202724b9" (UID: "cfba2c2e-4d57-4e22-8d75-1154202724b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:28.569954 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.569930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfba2c2e-4d57-4e22-8d75-1154202724b9" (UID: "cfba2c2e-4d57-4e22-8d75-1154202724b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:28.581106 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.581080    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfba2c2e-4d57-4e22-8d75-1154202724b9" (UID: "cfba2c2e-4d57-4e22-8d75-1154202724b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:28.651335 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.651307    2580 generic.go:356] "Generic (PLEG): container finished" podID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerID="f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431" exitCode=0
Sep 16 19:43:28.651450 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.651361    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"
Sep 16 19:43:28.651450 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.651365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" event={"ID":"cfba2c2e-4d57-4e22-8d75-1154202724b9","Type":"ContainerDied","Data":"f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431"}
Sep 16 19:43:28.651533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.651487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs" event={"ID":"cfba2c2e-4d57-4e22-8d75-1154202724b9","Type":"ContainerDied","Data":"1ead6a270056a65f0da23f6638af2814907d7e12efc5a192be7fb40dd915f055"}
Sep 16 19:43:28.651533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.651524    2580 scope.go:117] "RemoveContainer" containerID="f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431"
Sep 16 19:43:28.659093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.659073    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqg6b\" (UniqueName: \"kubernetes.io/projected/cfba2c2e-4d57-4e22-8d75-1154202724b9-kube-api-access-fqg6b\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:43:28.659171 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.659101    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:43:28.659171 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.659116    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:43:28.659171 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.659129    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:43:28.659171 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.659140    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfba2c2e-4d57-4e22-8d75-1154202724b9-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:43:28.659171 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.659151    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfba2c2e-4d57-4e22-8d75-1154202724b9-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:43:28.660729 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.660713    2580 scope.go:117] "RemoveContainer" containerID="1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424"
Sep 16 19:43:28.668689 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.668673    2580 scope.go:117] "RemoveContainer" containerID="f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431"
Sep 16 19:43:28.668970 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:43:28.668951    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431\": container with ID starting with f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431 not found: ID does not exist" containerID="f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431"
Sep 16 19:43:28.669024 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.668977    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431"} err="failed to get container status \"f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431\": rpc error: code = NotFound desc = could not find container \"f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431\": container with ID starting with f79bb0dfb3e08f0803f273399d19359a3f39ca762608c0da20b1f913a3bc0431 not found: ID does not exist"
Sep 16 19:43:28.669024 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.668995    2580 scope.go:117] "RemoveContainer" containerID="1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424"
Sep 16 19:43:28.669249 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:43:28.669232    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424\": container with ID starting with 1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424 not found: ID does not exist" containerID="1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424"
Sep 16 19:43:28.669294 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.669255    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424"} err="failed to get container status \"1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424\": rpc error: code = NotFound desc = could not find container \"1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424\": container with ID starting with 1100613c072764a9d7f567898517e8c4d67ebecad0845a588968a27e02062424 not found: ID does not exist"
Sep 16 19:43:28.673429 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.673408    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"]
Sep 16 19:43:28.677545 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:28.677527    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7b545f4984-ng6gs"]
Sep 16 19:43:30.276537 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:43:30.276506    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" path="/var/lib/kubelet/pods/cfba2c2e-4d57-4e22-8d75-1154202724b9/volumes"
Sep 16 19:44:26.221223 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.221187    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"]
Sep 16 19:44:26.221810 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.221543    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="storage-initializer"
Sep 16 19:44:26.221810 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.221555    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="storage-initializer"
Sep 16 19:44:26.221810 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.221566    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main"
Sep 16 19:44:26.221810 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.221572    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main"
Sep 16 19:44:26.221810 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.221629    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfba2c2e-4d57-4e22-8d75-1154202724b9" containerName="main"
Sep 16 19:44:26.224616 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.224597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.227053 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.227031    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:44:26.227211 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.227189    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 16 19:44:26.227283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.227207    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 16 19:44:26.227283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.227212    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-rl8ks\""
Sep 16 19:44:26.227391 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.227133    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 16 19:44:26.227391 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.227065    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 16 19:44:26.238288 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.238270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"]
Sep 16 19:44:26.296260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.296231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tmp-dir\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.296408 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.296276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-model-cache\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.296408 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.296290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-cabundle-cert\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.296408 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.296337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.296408 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.296367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-home\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.296553 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.296444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-dshm\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.296553 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.296480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdp6\" (UniqueName: \"kubernetes.io/projected/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kube-api-access-7gdp6\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.296553 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.296504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tls-certs\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397213 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-dshm\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397357 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdp6\" (UniqueName: \"kubernetes.io/projected/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kube-api-access-7gdp6\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397357 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tls-certs\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397357 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tmp-dir\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397357 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-model-cache\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397561 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-cabundle-cert\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397561 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397561 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-home\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397736 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-model-cache\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397802 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tmp-dir\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397933 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-home\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.397933 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.397836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.398062 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.398043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-cabundle-cert\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.400066 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.400047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-dshm\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.400367 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.400347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tls-certs\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.405700 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.405678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdp6\" (UniqueName: \"kubernetes.io/projected/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kube-api-access-7gdp6\") pod \"lora-single-adapter-test-kserve-648b447df4-k8p6x\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.538306 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.538238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:26.815857 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.815833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"]
Sep 16 19:44:26.911290 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.911261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" event={"ID":"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c","Type":"ContainerStarted","Data":"55f2f42f94967a43ec78efa0e3fa2ccb3c5ab21bbb056e2c7ad1509f27c31ef6"}
Sep 16 19:44:26.911455 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:26.911312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" event={"ID":"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c","Type":"ContainerStarted","Data":"4e7340f80424298ac94262eab0c19d3b01bb96503838ce82b643a4cdc72a53c7"}
Sep 16 19:44:29.928265 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:29.928228    2580 generic.go:356] "Generic (PLEG): container finished" podID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerID="55f2f42f94967a43ec78efa0e3fa2ccb3c5ab21bbb056e2c7ad1509f27c31ef6" exitCode=0
Sep 16 19:44:29.928627 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:29.928297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" event={"ID":"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c","Type":"ContainerDied","Data":"55f2f42f94967a43ec78efa0e3fa2ccb3c5ab21bbb056e2c7ad1509f27c31ef6"}
Sep 16 19:44:30.935310 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:30.935279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" event={"ID":"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c","Type":"ContainerStarted","Data":"23ff670c42a372eece26fa283b9105d53b4b858cc2c0b9eafc692d3f67ea6edb"}
Sep 16 19:44:30.965287 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:30.965242    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podStartSLOduration=2.377291317 podStartE2EDuration="4.965228577s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.58793726s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:44:30.964514591 +0000 UTC m=+3651.226191614" watchObservedRunningTime="2026-09-16 19:44:30.965228577 +0000 UTC m=+3651.226905602"
Sep 16 19:44:31.539095 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:31.539063    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:36.539020 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:36.538938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:44:36.540451 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:36.540421    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:44:46.538720 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:46.538664    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:44:52.112556 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:52.112516    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"]
Sep 16 19:44:56.539257 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:56.539204    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:44:59.468684 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:44:59.468644    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" podUID="28b3aa3c-6c8e-4c13-822f-ea770478f845" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:06.539588 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:06.539490    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:45:07.139697 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:07.139658    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" podUID="28b3aa3c-6c8e-4c13-822f-ea770478f845" containerName="main" containerID="cri-o://9a3df2c3e799ef8626ed8797be9d7d2a5f09997bedc173c3ab6607ffe5332a6d" gracePeriod=45
Sep 16 19:45:09.468630 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:09.468591    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" podUID="28b3aa3c-6c8e-4c13-822f-ea770478f845" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:16.539058 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:16.539019    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:45:19.468873 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:19.468831    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" podUID="28b3aa3c-6c8e-4c13-822f-ea770478f845" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:19.469262 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:19.468958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:45:26.539147 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:26.539103    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:45:29.469098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:29.469063    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" podUID="28b3aa3c-6c8e-4c13-822f-ea770478f845" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:36.538721 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:36.538624    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:45:37.260658 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:37.260627    2580 generic.go:356] "Generic (PLEG): container finished" podID="28b3aa3c-6c8e-4c13-822f-ea770478f845" containerID="9a3df2c3e799ef8626ed8797be9d7d2a5f09997bedc173c3ab6607ffe5332a6d" exitCode=0
Sep 16 19:45:37.260783 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:37.260671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" event={"ID":"28b3aa3c-6c8e-4c13-822f-ea770478f845","Type":"ContainerDied","Data":"9a3df2c3e799ef8626ed8797be9d7d2a5f09997bedc173c3ab6607ffe5332a6d"}
Sep 16 19:45:37.412269 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:37.412212    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:45:37.507219 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:37.507189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b3aa3c-6c8e-4c13-822f-ea770478f845-tls-certs\") pod \"28b3aa3c-6c8e-4c13-822f-ea770478f845\" (UID: \"28b3aa3c-6c8e-4c13-822f-ea770478f845\") "
Sep 16 19:45:37.507355 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:37.507234    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg7dt\" (UniqueName: \"kubernetes.io/projected/28b3aa3c-6c8e-4c13-822f-ea770478f845-kube-api-access-fg7dt\") pod \"28b3aa3c-6c8e-4c13-822f-ea770478f845\" (UID: \"28b3aa3c-6c8e-4c13-822f-ea770478f845\") "
Sep 16 19:45:37.509479 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:37.509445    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b3aa3c-6c8e-4c13-822f-ea770478f845-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28b3aa3c-6c8e-4c13-822f-ea770478f845" (UID: "28b3aa3c-6c8e-4c13-822f-ea770478f845"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:37.509580 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:37.509481    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b3aa3c-6c8e-4c13-822f-ea770478f845-kube-api-access-fg7dt" (OuterVolumeSpecName: "kube-api-access-fg7dt") pod "28b3aa3c-6c8e-4c13-822f-ea770478f845" (UID: "28b3aa3c-6c8e-4c13-822f-ea770478f845"). InnerVolumeSpecName "kube-api-access-fg7dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:37.607999 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:37.607970    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg7dt\" (UniqueName: \"kubernetes.io/projected/28b3aa3c-6c8e-4c13-822f-ea770478f845-kube-api-access-fg7dt\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:45:37.607999 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:37.608001    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b3aa3c-6c8e-4c13-822f-ea770478f845-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:45:38.266855 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:38.266821    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"
Sep 16 19:45:38.266855 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:38.266830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv" event={"ID":"28b3aa3c-6c8e-4c13-822f-ea770478f845","Type":"ContainerDied","Data":"30ab78a649032f0dab5c6893ca5ed04ba0d0500f85f1d6233d858112197a6ce1"}
Sep 16 19:45:38.267132 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:38.266946    2580 scope.go:117] "RemoveContainer" containerID="9a3df2c3e799ef8626ed8797be9d7d2a5f09997bedc173c3ab6607ffe5332a6d"
Sep 16 19:45:38.289484 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:38.289455    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"]
Sep 16 19:45:38.293438 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:38.293412    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58xhrwv"]
Sep 16 19:45:40.276140 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:40.276104    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28b3aa3c-6c8e-4c13-822f-ea770478f845" path="/var/lib/kubelet/pods/28b3aa3c-6c8e-4c13-822f-ea770478f845/volumes"
Sep 16 19:45:42.516876 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.516851    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"]
Sep 16 19:45:42.517286 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.517271    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b3aa3c-6c8e-4c13-822f-ea770478f845" containerName="main"
Sep 16 19:45:42.517335 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.517288    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b3aa3c-6c8e-4c13-822f-ea770478f845" containerName="main"
Sep 16 19:45:42.517374 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.517349    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b3aa3c-6c8e-4c13-822f-ea770478f845" containerName="main"
Sep 16 19:45:42.521567 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.521548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.524690 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.524666    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:45:42.524803 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.524743    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-8krqr\""
Sep 16 19:45:42.524803 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.524756    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:45:42.524965 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.524818    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:45:42.537115 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.537092    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"]
Sep 16 19:45:42.647531 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.647499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcbhq\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-kube-api-access-xcbhq\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.647688 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.647547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.647688 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.647574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.647688 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.647626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.647793 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.647689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.647793 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.647705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.647793 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.647720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.647793 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.647737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.647793 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.647751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.748994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.748966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.748994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.748992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcbhq\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-kube-api-access-xcbhq\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749561 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749561 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749628 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749628 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.749939 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.749920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.751485 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.751464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.751801 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.751781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.756611 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.756592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.756884 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.756864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcbhq\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-kube-api-access-xcbhq\") pod \"router-gateway-1-openshift-default-56bd58cf4d-r7shp\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:42.833305 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:42.833252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:43.108622 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:43.108535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"]
Sep 16 19:45:43.111868 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:45:43.111843    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf09fcfa_e066_4cb6_b089_72840c3a4fa0.slice/crio-3ad91e4267dacb3dbbe7c583fb70d8f5f7af608cbb73327b792a0607c4c7c0fb WatchSource:0}: Error finding container 3ad91e4267dacb3dbbe7c583fb70d8f5f7af608cbb73327b792a0607c4c7c0fb: Status 404 returned error can't find the container with id 3ad91e4267dacb3dbbe7c583fb70d8f5f7af608cbb73327b792a0607c4c7c0fb
Sep 16 19:45:43.113826 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:43.113791    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:45:43.113883 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:43.113867    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:45:43.113941 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:43.113918    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:45:43.294071 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:43.294035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" event={"ID":"bf09fcfa-e066-4cb6-b089-72840c3a4fa0","Type":"ContainerStarted","Data":"e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12"}
Sep 16 19:45:43.294261 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:43.294096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" event={"ID":"bf09fcfa-e066-4cb6-b089-72840c3a4fa0","Type":"ContainerStarted","Data":"3ad91e4267dacb3dbbe7c583fb70d8f5f7af608cbb73327b792a0607c4c7c0fb"}
Sep 16 19:45:43.313123 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:43.313032    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" podStartSLOduration=1.3130117989999999 podStartE2EDuration="1.313011799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:43.312102902 +0000 UTC m=+3723.573779914" watchObservedRunningTime="2026-09-16 19:45:43.313011799 +0000 UTC m=+3723.574688811"
Sep 16 19:45:43.833532 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:43.833501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:44.838841 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:44.838806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:45.303645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:45.303612    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:45.304483 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:45.304462    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:46.553472 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:46.553442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:45:46.561470 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:46.561432    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:45:55.348107 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:55.348075    2580 generic.go:356] "Generic (PLEG): container finished" podID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerID="e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12" exitCode=0
Sep 16 19:45:55.348485 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:55.348140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" event={"ID":"bf09fcfa-e066-4cb6-b089-72840c3a4fa0","Type":"ContainerDied","Data":"e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12"}
Sep 16 19:45:55.348485 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:55.348480    2580 scope.go:117] "RemoveContainer" containerID="e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12"
Sep 16 19:45:55.349422 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:55.349398    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:45:55.349489 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:55.349459    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:45:55.349539 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:55.349491    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:45:56.361252 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:56.361204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" event={"ID":"bf09fcfa-e066-4cb6-b089-72840c3a4fa0","Type":"ContainerStarted","Data":"d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf"}
Sep 16 19:45:56.833968 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:56.833939    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:45:56.834874 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:56.834849    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 19:45:57.834283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:57.834243    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 19:45:58.834096 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:58.834050    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 19:45:59.834118 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:45:59.834073    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 19:46:00.303790 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:00.303744    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:46:00.834076 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:00.834037    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 16 19:46:01.838018 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:01.837987    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:46:01.839004 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:01.838963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:46:37.754312 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:37.754283    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-648b447df4-k8p6x_2da6e91b-93d1-4ef9-905b-2f906c5b9b3c/storage-initializer/0.log"
Sep 16 19:46:37.771136 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:37.771114    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-648b447df4-k8p6x_2da6e91b-93d1-4ef9-905b-2f906c5b9b3c/main/0.log"
Sep 16 19:46:39.112729 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:39.112695    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"]
Sep 16 19:46:54.134546 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.134499    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main" containerID="cri-o://23ff670c42a372eece26fa283b9105d53b4b858cc2c0b9eafc692d3f67ea6edb" gracePeriod=45
Sep 16 19:46:54.554375 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.554328    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:46:54.655438 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.655399    2580 generic.go:356] "Generic (PLEG): container finished" podID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerID="23ff670c42a372eece26fa283b9105d53b4b858cc2c0b9eafc692d3f67ea6edb" exitCode=0
Sep 16 19:46:54.655538 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.655464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" event={"ID":"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c","Type":"ContainerDied","Data":"23ff670c42a372eece26fa283b9105d53b4b858cc2c0b9eafc692d3f67ea6edb"}
Sep 16 19:46:54.791126 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.791100    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:46:54.893703 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.893602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-model-cache\") pod \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") "
Sep 16 19:46:54.893703 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.893673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gdp6\" (UniqueName: \"kubernetes.io/projected/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kube-api-access-7gdp6\") pod \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") "
Sep 16 19:46:54.893994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.893731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tls-certs\") pod \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") "
Sep 16 19:46:54.893994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.893757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-home\") pod \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") "
Sep 16 19:46:54.893994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.893777    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tmp-dir\") pod \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") "
Sep 16 19:46:54.893994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.893806    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-cabundle-cert\") pod \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") "
Sep 16 19:46:54.893994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.893829    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-dshm\") pod \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") "
Sep 16 19:46:54.893994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.893874    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kserve-provision-location\") pod \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\" (UID: \"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c\") "
Sep 16 19:46:54.893994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.893956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" (UID: "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:54.894365 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.894174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-home" (OuterVolumeSpecName: "home") pod "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" (UID: "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:54.894365 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.894200    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" (UID: "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:46:54.894365 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.894264    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:46:54.896304 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.896277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kube-api-access-7gdp6" (OuterVolumeSpecName: "kube-api-access-7gdp6") pod "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" (UID: "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c"). InnerVolumeSpecName "kube-api-access-7gdp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:54.896692 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.896670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-dshm" (OuterVolumeSpecName: "dshm") pod "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" (UID: "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:54.896760 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.896673    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" (UID: "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:46:54.905567 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.905539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" (UID: "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:54.919178 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.919147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" (UID: "2da6e91b-93d1-4ef9-905b-2f906c5b9b3c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:54.995628 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.995604    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:46:54.995750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.995631    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:46:54.995750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.995645    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:46:54.995750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.995657    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:46:54.995750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.995668    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:46:54.995750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.995681    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:46:54.995750 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:54.995694    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gdp6\" (UniqueName: \"kubernetes.io/projected/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c-kube-api-access-7gdp6\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:46:55.662511 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:55.662464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x" event={"ID":"2da6e91b-93d1-4ef9-905b-2f906c5b9b3c","Type":"ContainerDied","Data":"4e7340f80424298ac94262eab0c19d3b01bb96503838ce82b643a4cdc72a53c7"}
Sep 16 19:46:55.662511 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:55.662502    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"
Sep 16 19:46:55.663040 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:55.662545    2580 scope.go:117] "RemoveContainer" containerID="23ff670c42a372eece26fa283b9105d53b4b858cc2c0b9eafc692d3f67ea6edb"
Sep 16 19:46:55.671994 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:55.671970    2580 scope.go:117] "RemoveContainer" containerID="55f2f42f94967a43ec78efa0e3fa2ccb3c5ab21bbb056e2c7ad1509f27c31ef6"
Sep 16 19:46:55.691383 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:55.691353    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"]
Sep 16 19:46:55.692933 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:55.692883    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-648b447df4-k8p6x"]
Sep 16 19:46:56.275934 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:46:56.275870    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" path="/var/lib/kubelet/pods/2da6e91b-93d1-4ef9-905b-2f906c5b9b3c/volumes"
Sep 16 19:47:03.323466 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.323432    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"]
Sep 16 19:47:03.324005 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.323903    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="storage-initializer"
Sep 16 19:47:03.324005 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.323921    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="storage-initializer"
Sep 16 19:47:03.324005 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.323958    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main"
Sep 16 19:47:03.324005 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.323976    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main"
Sep 16 19:47:03.324260 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.324098    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da6e91b-93d1-4ef9-905b-2f906c5b9b3c" containerName="main"
Sep 16 19:47:03.328942 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.328920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.331194 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.331174    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 16 19:47:03.331357 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.331342    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 16 19:47:03.331462 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.331438    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:47:03.331526 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.331459    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 16 19:47:03.331526 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.331461    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 16 19:47:03.331650 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.331575    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-wmpp6\""
Sep 16 19:47:03.340859 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.340835    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"]
Sep 16 19:47:03.364769 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.364735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-home\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.364942 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.364771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-dshm\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.364942 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.364797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7g4\" (UniqueName: \"kubernetes.io/projected/c2caffa1-891a-497c-ad21-2d09419ca2c8-kube-api-access-ct7g4\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.364942 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.364864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.364942 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.364933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-model-cache\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.365098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.364951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.365098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.364967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2caffa1-891a-497c-ad21-2d09419ca2c8-tls-certs\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.365098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.364987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2caffa1-891a-497c-ad21-2d09419ca2c8-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.465844 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.465806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-model-cache\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.465844 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.465844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466112 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.465860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2caffa1-891a-497c-ad21-2d09419ca2c8-tls-certs\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466112 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.465882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2caffa1-891a-497c-ad21-2d09419ca2c8-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466112 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.465976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-home\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466112 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.465991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-dshm\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466112 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.466016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7g4\" (UniqueName: \"kubernetes.io/projected/c2caffa1-891a-497c-ad21-2d09419ca2c8-kube-api-access-ct7g4\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466112 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.466078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466434 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.466339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466434 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.466349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-model-cache\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466544 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.466476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466669 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.466650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-home\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.466740 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.466685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2caffa1-891a-497c-ad21-2d09419ca2c8-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.468566 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.468543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-dshm\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.468786 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.468766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2caffa1-891a-497c-ad21-2d09419ca2c8-tls-certs\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.474037 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.474016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7g4\" (UniqueName: \"kubernetes.io/projected/c2caffa1-891a-497c-ad21-2d09419ca2c8-kube-api-access-ct7g4\") pod \"lora-multiple-adapters-test-kserve-695ddf77b9-pskbh\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.640161 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.640073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:03.919381 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:03.919353    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"]
Sep 16 19:47:03.921973 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:47:03.921944    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2caffa1_891a_497c_ad21_2d09419ca2c8.slice/crio-547cafefb8aa121d62e03413286fe633997ee992570a075aac7d6f82f31af252 WatchSource:0}: Error finding container 547cafefb8aa121d62e03413286fe633997ee992570a075aac7d6f82f31af252: Status 404 returned error can't find the container with id 547cafefb8aa121d62e03413286fe633997ee992570a075aac7d6f82f31af252
Sep 16 19:47:04.709681 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:04.709637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" event={"ID":"c2caffa1-891a-497c-ad21-2d09419ca2c8","Type":"ContainerStarted","Data":"1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f"}
Sep 16 19:47:04.710215 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:04.709706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" event={"ID":"c2caffa1-891a-497c-ad21-2d09419ca2c8","Type":"ContainerStarted","Data":"547cafefb8aa121d62e03413286fe633997ee992570a075aac7d6f82f31af252"}
Sep 16 19:47:07.724257 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:07.724221    2580 generic.go:356] "Generic (PLEG): container finished" podID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerID="1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f" exitCode=0
Sep 16 19:47:07.724734 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:07.724294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" event={"ID":"c2caffa1-891a-497c-ad21-2d09419ca2c8","Type":"ContainerDied","Data":"1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f"}
Sep 16 19:47:08.730730 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:08.730686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" event={"ID":"c2caffa1-891a-497c-ad21-2d09419ca2c8","Type":"ContainerStarted","Data":"8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53"}
Sep 16 19:47:08.756863 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:08.756801    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podStartSLOduration=2.821859668 podStartE2EDuration="5.756781224s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.934921556s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:08.755509887 +0000 UTC m=+3809.017186899" watchObservedRunningTime="2026-09-16 19:47:08.756781224 +0000 UTC m=+3809.018458236"
Sep 16 19:47:09.641108 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:09.641064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:13.640829 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:13.640786    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:47:13.642472 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:13.642439    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 16 19:47:23.641492 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:23.641449    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 16 19:47:33.640695 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:33.640639    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 16 19:47:43.640731 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:43.640680    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 16 19:47:53.640704 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:47:53.640659    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 16 19:48:03.641515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:48:03.641407    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 16 19:48:13.641090 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:48:13.641040    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 16 19:48:23.650571 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:48:23.650542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:48:23.658628 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:48:23.658601    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:48:49.547774 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:48:49.547744    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-695ddf77b9-pskbh_c2caffa1-891a-497c-ad21-2d09419ca2c8/storage-initializer/0.log"
Sep 16 19:48:49.561139 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:48:49.561111    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-695ddf77b9-pskbh_c2caffa1-891a-497c-ad21-2d09419ca2c8/main/0.log"
Sep 16 19:48:50.547675 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:48:50.547636    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"]
Sep 16 19:48:58.214287 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:48:58.214247    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"]
Sep 16 19:48:58.214771 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:48:58.214464    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy" containerID="cri-o://d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf" gracePeriod=30
Sep 16 19:49:01.841510 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:01.841463    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:49:03.463565 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.463538    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:49:03.555852 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.555810    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-certs\") pod \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") "
Sep 16 19:49:03.556064 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.555880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-socket\") pod \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") "
Sep 16 19:49:03.556064 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.555939    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-data\") pod \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") "
Sep 16 19:49:03.556064 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.555971    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-credential-socket\") pod \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") "
Sep 16 19:49:03.556064 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcbhq\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-kube-api-access-xcbhq\") pod \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") "
Sep 16 19:49:03.556064 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-envoy\") pod \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") "
Sep 16 19:49:03.556273 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556091    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-podinfo\") pod \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") "
Sep 16 19:49:03.556273 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-token\") pod \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") "
Sep 16 19:49:03.556273 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istiod-ca-cert\") pod \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\" (UID: \"bf09fcfa-e066-4cb6-b089-72840c3a4fa0\") "
Sep 16 19:49:03.556273 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf09fcfa-e066-4cb6-b089-72840c3a4fa0" (UID: "bf09fcfa-e066-4cb6-b089-72840c3a4fa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:03.556273 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf09fcfa-e066-4cb6-b089-72840c3a4fa0" (UID: "bf09fcfa-e066-4cb6-b089-72840c3a4fa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:03.556530 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf09fcfa-e066-4cb6-b089-72840c3a4fa0" (UID: "bf09fcfa-e066-4cb6-b089-72840c3a4fa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:03.556530 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556361    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:03.556530 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556383    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-data\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:03.556530 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556394    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-workload-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:03.556742 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf09fcfa-e066-4cb6-b089-72840c3a4fa0" (UID: "bf09fcfa-e066-4cb6-b089-72840c3a4fa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:49:03.556742 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.556722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf09fcfa-e066-4cb6-b089-72840c3a4fa0" (UID: "bf09fcfa-e066-4cb6-b089-72840c3a4fa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:03.558646 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.558620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf09fcfa-e066-4cb6-b089-72840c3a4fa0" (UID: "bf09fcfa-e066-4cb6-b089-72840c3a4fa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:49:03.559076 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.559036    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf09fcfa-e066-4cb6-b089-72840c3a4fa0" (UID: "bf09fcfa-e066-4cb6-b089-72840c3a4fa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:03.559076 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.559064    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-kube-api-access-xcbhq" (OuterVolumeSpecName: "kube-api-access-xcbhq") pod "bf09fcfa-e066-4cb6-b089-72840c3a4fa0" (UID: "bf09fcfa-e066-4cb6-b089-72840c3a4fa0"). InnerVolumeSpecName "kube-api-access-xcbhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:03.559216 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.559119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf09fcfa-e066-4cb6-b089-72840c3a4fa0" (UID: "bf09fcfa-e066-4cb6-b089-72840c3a4fa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:03.657275 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.657240    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istiod-ca-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:03.657275 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.657268    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-credential-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:03.657275 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.657277    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcbhq\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-kube-api-access-xcbhq\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:03.657515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.657287    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-envoy\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:03.657515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.657298    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-podinfo\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:03.657515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:03.657305    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf09fcfa-e066-4cb6-b089-72840c3a4fa0-istio-token\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:04.315180 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.315147    2580 generic.go:356] "Generic (PLEG): container finished" podID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerID="d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf" exitCode=0
Sep 16 19:49:04.315356 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.315193    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"
Sep 16 19:49:04.315356 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.315204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" event={"ID":"bf09fcfa-e066-4cb6-b089-72840c3a4fa0","Type":"ContainerDied","Data":"d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf"}
Sep 16 19:49:04.315356 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.315258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp" event={"ID":"bf09fcfa-e066-4cb6-b089-72840c3a4fa0","Type":"ContainerDied","Data":"3ad91e4267dacb3dbbe7c583fb70d8f5f7af608cbb73327b792a0607c4c7c0fb"}
Sep 16 19:49:04.315356 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.315287    2580 scope.go:117] "RemoveContainer" containerID="d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf"
Sep 16 19:49:04.324328 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.324310    2580 scope.go:117] "RemoveContainer" containerID="e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12"
Sep 16 19:49:04.332382 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.332345    2580 scope.go:117] "RemoveContainer" containerID="d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf"
Sep 16 19:49:04.332676 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:49:04.332654    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf\": container with ID starting with d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf not found: ID does not exist" containerID="d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf"
Sep 16 19:49:04.332759 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.332685    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf"} err="failed to get container status \"d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf\": rpc error: code = NotFound desc = could not find container \"d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf\": container with ID starting with d858532e39690a6bd0da99300b2aafc3727a72f211359a6b5e92ae7b304693bf not found: ID does not exist"
Sep 16 19:49:04.332759 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.332706    2580 scope.go:117] "RemoveContainer" containerID="e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12"
Sep 16 19:49:04.333016 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:49:04.332995    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12\": container with ID starting with e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12 not found: ID does not exist" containerID="e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12"
Sep 16 19:49:04.333093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.333023    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12"} err="failed to get container status \"e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12\": rpc error: code = NotFound desc = could not find container \"e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12\": container with ID starting with e83fa2a43dc05758259b739a480918146bfdac3fb7cd6e3ceb9592a0a3e41e12 not found: ID does not exist"
Sep 16 19:49:04.333672 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.333650    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"]
Sep 16 19:49:04.337359 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:04.337338    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-r7shp"]
Sep 16 19:49:05.570492 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:05.570440    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main" containerID="cri-o://8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53" gracePeriod=45
Sep 16 19:49:05.651496 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:05.651449    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 16 19:49:06.231212 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.231186    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:49:06.276209 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.276174    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" path="/var/lib/kubelet/pods/bf09fcfa-e066-4cb6-b089-72840c3a4fa0/volumes"
Sep 16 19:49:06.327858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.327760    2580 generic.go:356] "Generic (PLEG): container finished" podID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerID="8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53" exitCode=0
Sep 16 19:49:06.327858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.327818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" event={"ID":"c2caffa1-891a-497c-ad21-2d09419ca2c8","Type":"ContainerDied","Data":"8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53"}
Sep 16 19:49:06.327858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.327828    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"
Sep 16 19:49:06.328137 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.327870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh" event={"ID":"c2caffa1-891a-497c-ad21-2d09419ca2c8","Type":"ContainerDied","Data":"547cafefb8aa121d62e03413286fe633997ee992570a075aac7d6f82f31af252"}
Sep 16 19:49:06.328137 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.327912    2580 scope.go:117] "RemoveContainer" containerID="8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53"
Sep 16 19:49:06.337085 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.337066    2580 scope.go:117] "RemoveContainer" containerID="1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f"
Sep 16 19:49:06.346304 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.346281    2580 scope.go:117] "RemoveContainer" containerID="8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53"
Sep 16 19:49:06.346610 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:49:06.346590    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53\": container with ID starting with 8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53 not found: ID does not exist" containerID="8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53"
Sep 16 19:49:06.346660 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.346620    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53"} err="failed to get container status \"8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53\": rpc error: code = NotFound desc = could not find container \"8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53\": container with ID starting with 8041f3962146f9f8d0af1ef9a8d349aaa935c531f083943095e6a41a49528d53 not found: ID does not exist"
Sep 16 19:49:06.346660 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.346640    2580 scope.go:117] "RemoveContainer" containerID="1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f"
Sep 16 19:49:06.346974 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:49:06.346942    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f\": container with ID starting with 1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f not found: ID does not exist" containerID="1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f"
Sep 16 19:49:06.347027 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.346986    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f"} err="failed to get container status \"1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f\": rpc error: code = NotFound desc = could not find container \"1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f\": container with ID starting with 1c3e17d280c189b9268847d1f0026f459deb97e510b55132bf07f8041d0b8d8f not found: ID does not exist"
Sep 16 19:49:06.383319 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.383282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct7g4\" (UniqueName: \"kubernetes.io/projected/c2caffa1-891a-497c-ad21-2d09419ca2c8-kube-api-access-ct7g4\") pod \"c2caffa1-891a-497c-ad21-2d09419ca2c8\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") "
Sep 16 19:49:06.383464 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.383330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2caffa1-891a-497c-ad21-2d09419ca2c8-cabundle-cert\") pod \"c2caffa1-891a-497c-ad21-2d09419ca2c8\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") "
Sep 16 19:49:06.383464 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.383370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-tmp-dir\") pod \"c2caffa1-891a-497c-ad21-2d09419ca2c8\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") "
Sep 16 19:49:06.383464 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.383411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-kserve-provision-location\") pod \"c2caffa1-891a-497c-ad21-2d09419ca2c8\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") "
Sep 16 19:49:06.383588 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.383501    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2caffa1-891a-497c-ad21-2d09419ca2c8-tls-certs\") pod \"c2caffa1-891a-497c-ad21-2d09419ca2c8\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") "
Sep 16 19:49:06.383588 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.383542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-model-cache\") pod \"c2caffa1-891a-497c-ad21-2d09419ca2c8\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") "
Sep 16 19:49:06.383588 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.383573    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-home\") pod \"c2caffa1-891a-497c-ad21-2d09419ca2c8\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") "
Sep 16 19:49:06.384275 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.383860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2caffa1-891a-497c-ad21-2d09419ca2c8" (UID: "c2caffa1-891a-497c-ad21-2d09419ca2c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:06.384275 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.383881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2caffa1-891a-497c-ad21-2d09419ca2c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c2caffa1-891a-497c-ad21-2d09419ca2c8" (UID: "c2caffa1-891a-497c-ad21-2d09419ca2c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:49:06.384275 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.384241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-home" (OuterVolumeSpecName: "home") pod "c2caffa1-891a-497c-ad21-2d09419ca2c8" (UID: "c2caffa1-891a-497c-ad21-2d09419ca2c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:06.384525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.384395    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-dshm\") pod \"c2caffa1-891a-497c-ad21-2d09419ca2c8\" (UID: \"c2caffa1-891a-497c-ad21-2d09419ca2c8\") "
Sep 16 19:49:06.387273 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.387242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2caffa1-891a-497c-ad21-2d09419ca2c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2caffa1-891a-497c-ad21-2d09419ca2c8" (UID: "c2caffa1-891a-497c-ad21-2d09419ca2c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:06.387407 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.387342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2caffa1-891a-497c-ad21-2d09419ca2c8-kube-api-access-ct7g4" (OuterVolumeSpecName: "kube-api-access-ct7g4") pod "c2caffa1-891a-497c-ad21-2d09419ca2c8" (UID: "c2caffa1-891a-497c-ad21-2d09419ca2c8"). InnerVolumeSpecName "kube-api-access-ct7g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:06.387911 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.387864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-dshm" (OuterVolumeSpecName: "dshm") pod "c2caffa1-891a-497c-ad21-2d09419ca2c8" (UID: "c2caffa1-891a-497c-ad21-2d09419ca2c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:06.395468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.390793    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2caffa1-891a-497c-ad21-2d09419ca2c8-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:06.395468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.392159    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:06.395468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.392185    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:06.395468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.392349    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:06.395468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.392363    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct7g4\" (UniqueName: \"kubernetes.io/projected/c2caffa1-891a-497c-ad21-2d09419ca2c8-kube-api-access-ct7g4\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:06.395468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.392377    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2caffa1-891a-497c-ad21-2d09419ca2c8-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:06.400980 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.400954    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2caffa1-891a-497c-ad21-2d09419ca2c8" (UID: "c2caffa1-891a-497c-ad21-2d09419ca2c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:06.414590 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.414552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2caffa1-891a-497c-ad21-2d09419ca2c8" (UID: "c2caffa1-891a-497c-ad21-2d09419ca2c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:06.493380 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.493343    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:06.493380 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.493375    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2caffa1-891a-497c-ad21-2d09419ca2c8-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:06.652148 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.652114    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"]
Sep 16 19:49:06.659946 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:06.659876    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-695ddf77b9-pskbh"]
Sep 16 19:49:08.276540 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:08.276497    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" path="/var/lib/kubelet/pods/c2caffa1-891a-497c-ad21-2d09419ca2c8/volumes"
Sep 16 19:49:10.811299 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811257    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"]
Sep 16 19:49:10.811766 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811623    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy"
Sep 16 19:49:10.811766 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811636    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy"
Sep 16 19:49:10.811766 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811648    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="storage-initializer"
Sep 16 19:49:10.811766 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811658    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="storage-initializer"
Sep 16 19:49:10.811766 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811672    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main"
Sep 16 19:49:10.811766 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811682    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main"
Sep 16 19:49:10.811766 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811769    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy"
Sep 16 19:49:10.812173 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811779    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2caffa1-891a-497c-ad21-2d09419ca2c8" containerName="main"
Sep 16 19:49:10.812173 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811786    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy"
Sep 16 19:49:10.812173 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811874    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy"
Sep 16 19:49:10.812173 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.811886    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf09fcfa-e066-4cb6-b089-72840c3a4fa0" containerName="istio-proxy"
Sep 16 19:49:10.816955 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.816934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:10.819209 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.819186    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:49:10.819337 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.819190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:49:10.819549 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.819533    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:49:10.819797 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.819783    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:49:10.819916 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.819839    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:49:10.819989 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.819962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-hsx46\""
Sep 16 19:49:10.838085 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.838064    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"]
Sep 16 19:49:10.936199 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.936164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:10.936199 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.936197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbc1ea-317d-4b98-966d-a3007461ded0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:10.936407 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.936272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:10.936407 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.936357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:10.936484 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.936415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:10.936484 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.936474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fbc1ea-317d-4b98-966d-a3007461ded0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:10.936550 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.936519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrzl\" (UniqueName: \"kubernetes.io/projected/e0fbc1ea-317d-4b98-966d-a3007461ded0-kube-api-access-skrzl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:10.936585 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:10.936565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037249 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037249 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037501 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fbc1ea-317d-4b98-966d-a3007461ded0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037501 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrzl\" (UniqueName: \"kubernetes.io/projected/e0fbc1ea-317d-4b98-966d-a3007461ded0-kube-api-access-skrzl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037501 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037501 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037501 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbc1ea-317d-4b98-966d-a3007461ded0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037501 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037832 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037832 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.037832 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.038030 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.037945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.038164 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.038144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbc1ea-317d-4b98-966d-a3007461ded0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.039773 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.039752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.039966 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.039949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fbc1ea-317d-4b98-966d-a3007461ded0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.051709 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.051677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrzl\" (UniqueName: \"kubernetes.io/projected/e0fbc1ea-317d-4b98-966d-a3007461ded0-kube-api-access-skrzl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.127703 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.127622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:11.413593 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:11.413509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"]
Sep 16 19:49:12.360235 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:12.360190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" event={"ID":"e0fbc1ea-317d-4b98-966d-a3007461ded0","Type":"ContainerStarted","Data":"47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e"}
Sep 16 19:49:12.360742 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:12.360259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" event={"ID":"e0fbc1ea-317d-4b98-966d-a3007461ded0","Type":"ContainerStarted","Data":"9ae1aaeb69cbc508e1a116df8a20636616a2b3e13374317bbce88a52a0f233fa"}
Sep 16 19:49:14.104363 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.104282    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"]
Sep 16 19:49:14.107698 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.107679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:14.109871 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.109845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 19:49:14.109871 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.109864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-wpcqv\""
Sep 16 19:49:14.110060 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.109874    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 19:49:14.110568 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.110550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 19:49:14.114519 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.114495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"]
Sep 16 19:49:14.269570 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.269530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxfl\" (UniqueName: \"kubernetes.io/projected/f1599806-fb1e-45d7-bf98-60cabec18df6-kube-api-access-wbxfl\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:14.370315 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.370228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxfl\" (UniqueName: \"kubernetes.io/projected/f1599806-fb1e-45d7-bf98-60cabec18df6-kube-api-access-wbxfl\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:14.371289 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.371263    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerID="47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e" exitCode=0
Sep 16 19:49:14.371403 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.371331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" event={"ID":"e0fbc1ea-317d-4b98-966d-a3007461ded0","Type":"ContainerDied","Data":"47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e"}
Sep 16 19:49:14.378955 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:14.378937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxfl\" (UniqueName: \"kubernetes.io/projected/f1599806-fb1e-45d7-bf98-60cabec18df6-kube-api-access-wbxfl\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:15.378350 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:15.378312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" event={"ID":"e0fbc1ea-317d-4b98-966d-a3007461ded0","Type":"ContainerStarted","Data":"c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6"}
Sep 16 19:49:15.402042 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:15.401974    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podStartSLOduration=3.076551471 podStartE2EDuration="5.401951566s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.325400095s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:15.398471752 +0000 UTC m=+3935.660148764" watchObservedRunningTime="2026-09-16 19:49:15.401951566 +0000 UTC m=+3935.663628580"
Sep 16 19:49:21.128227 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:21.128187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:21.128770 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:21.128262    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:49:21.129780 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:21.129747    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:49:22.232293 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:22.232258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:22.238022 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:22.237996    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:22.332996 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:22.332951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:22.333178 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:22.333075    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:22.336297 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:22.336274    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") DevicePath \"csi-0cc5978c3fe2779443ebbc8f452617873da5dd437517ac792bf72ab6bb84e5d2\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:22.684291 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:22.684256    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d9afeddae0d033bfca5c3cae2353cb957b5c56e40c32d6e74ff955208c9fa6f1/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:22.700425 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:22.700385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"e2e-pvc-lora-download-2mfgr\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:22.817691 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:22.817654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:23.094522 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:23.094434    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"]
Sep 16 19:49:23.096780 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:49:23.096750    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1599806_fb1e_45d7_bf98_60cabec18df6.slice/crio-2bdc5212231343d6475617ed48aeff3d76942550e6dca5f2b64ffd8314f85f07 WatchSource:0}: Error finding container 2bdc5212231343d6475617ed48aeff3d76942550e6dca5f2b64ffd8314f85f07: Status 404 returned error can't find the container with id 2bdc5212231343d6475617ed48aeff3d76942550e6dca5f2b64ffd8314f85f07
Sep 16 19:49:23.421953 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:23.421910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr" event={"ID":"f1599806-fb1e-45d7-bf98-60cabec18df6","Type":"ContainerStarted","Data":"616dddba15b0985938a60a47452dea16138b682cd7af9f8d22625cf854550f13"}
Sep 16 19:49:23.422328 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:23.421967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr" event={"ID":"f1599806-fb1e-45d7-bf98-60cabec18df6","Type":"ContainerStarted","Data":"2bdc5212231343d6475617ed48aeff3d76942550e6dca5f2b64ffd8314f85f07"}
Sep 16 19:49:23.437594 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:23.437530    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr" podStartSLOduration=12.437510749 podStartE2EDuration="12.437510749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:23.436365564 +0000 UTC m=+3943.698042576" watchObservedRunningTime="2026-09-16 19:49:23.437510749 +0000 UTC m=+3943.699187760"
Sep 16 19:49:25.435568 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:25.435531    2580 generic.go:356] "Generic (PLEG): container finished" podID="f1599806-fb1e-45d7-bf98-60cabec18df6" containerID="616dddba15b0985938a60a47452dea16138b682cd7af9f8d22625cf854550f13" exitCode=0
Sep 16 19:49:25.435568 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:25.435579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr" event={"ID":"f1599806-fb1e-45d7-bf98-60cabec18df6","Type":"ContainerDied","Data":"616dddba15b0985938a60a47452dea16138b682cd7af9f8d22625cf854550f13"}
Sep 16 19:49:26.594371 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:26.594341    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:26.775348 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:26.775246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbxfl\" (UniqueName: \"kubernetes.io/projected/f1599806-fb1e-45d7-bf98-60cabec18df6-kube-api-access-wbxfl\") pod \"f1599806-fb1e-45d7-bf98-60cabec18df6\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") "
Sep 16 19:49:26.775525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:26.775426    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") pod \"f1599806-fb1e-45d7-bf98-60cabec18df6\" (UID: \"f1599806-fb1e-45d7-bf98-60cabec18df6\") "
Sep 16 19:49:26.777679 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:26.777647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1599806-fb1e-45d7-bf98-60cabec18df6-kube-api-access-wbxfl" (OuterVolumeSpecName: "kube-api-access-wbxfl") pod "f1599806-fb1e-45d7-bf98-60cabec18df6" (UID: "f1599806-fb1e-45d7-bf98-60cabec18df6"). InnerVolumeSpecName "kube-api-access-wbxfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:26.779993 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:26.779955    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb" (OuterVolumeSpecName: "model-storage") pod "f1599806-fb1e-45d7-bf98-60cabec18df6" (UID: "f1599806-fb1e-45d7-bf98-60cabec18df6"). InnerVolumeSpecName "pvc-9245be29-5793-431d-9414-32dff13f3f9b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:49:26.876044 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:26.876001    2580 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") on node \"ip-10-0-139-95.ec2.internal\" "
Sep 16 19:49:26.876044 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:26.876032    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbxfl\" (UniqueName: \"kubernetes.io/projected/f1599806-fb1e-45d7-bf98-60cabec18df6-kube-api-access-wbxfl\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:49:27.446924 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:27.446879    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"
Sep 16 19:49:27.447085 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:27.446866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr" event={"ID":"f1599806-fb1e-45d7-bf98-60cabec18df6","Type":"ContainerDied","Data":"2bdc5212231343d6475617ed48aeff3d76942550e6dca5f2b64ffd8314f85f07"}
Sep 16 19:49:27.447085 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:27.447022    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bdc5212231343d6475617ed48aeff3d76942550e6dca5f2b64ffd8314f85f07"
Sep 16 19:49:28.129125 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:28.129104    2580 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-9245be29-5793-431d-9414-32dff13f3f9b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb") on node "ip-10-0-139-95.ec2.internal"
Sep 16 19:49:28.186475 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:28.186421    2580 reconciler_common.go:299] "Volume detached for volume \"pvc-9245be29-5793-431d-9414-32dff13f3f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bd3b235cd00981bb\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"csi-0cc5978c3fe2779443ebbc8f452617873da5dd437517ac792bf72ab6bb84e5d2\""
Sep 16 19:49:31.129054 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:31.128952    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:49:41.129049 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:41.128933    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:49:51.128796 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:49:51.128749    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:50:01.128772 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:50:01.128724    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:50:11.129157 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:50:11.129099    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:50:21.128435 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:50:21.128362    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:50:31.128498 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:50:31.128450    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:50:41.128423 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:50:41.128361    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:50:51.128977 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:50:51.128918    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:51:01.128525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:01.128424    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:51:11.128930 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:11.128863    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:51:21.139174 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:21.139136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:51:21.147480 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:21.147448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:51:35.195242 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:35.195206    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"]
Sep 16 19:51:50.219908 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:50.219824    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main" containerID="cri-o://c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6" gracePeriod=45
Sep 16 19:51:50.966077 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:50.966049    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:51:51.036281 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036187    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-tmp-dir\") pod \"e0fbc1ea-317d-4b98-966d-a3007461ded0\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") "
Sep 16 19:51:51.036281 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036234    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-dshm\") pod \"e0fbc1ea-317d-4b98-966d-a3007461ded0\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") "
Sep 16 19:51:51.036281 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036275    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbc1ea-317d-4b98-966d-a3007461ded0-cabundle-cert\") pod \"e0fbc1ea-317d-4b98-966d-a3007461ded0\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") "
Sep 16 19:51:51.036589 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-model-cache\") pod \"e0fbc1ea-317d-4b98-966d-a3007461ded0\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") "
Sep 16 19:51:51.036589 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036363    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skrzl\" (UniqueName: \"kubernetes.io/projected/e0fbc1ea-317d-4b98-966d-a3007461ded0-kube-api-access-skrzl\") pod \"e0fbc1ea-317d-4b98-966d-a3007461ded0\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") "
Sep 16 19:51:51.036589 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036481    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fbc1ea-317d-4b98-966d-a3007461ded0-tls-certs\") pod \"e0fbc1ea-317d-4b98-966d-a3007461ded0\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") "
Sep 16 19:51:51.036589 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-kserve-provision-location\") pod \"e0fbc1ea-317d-4b98-966d-a3007461ded0\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") "
Sep 16 19:51:51.036806 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036628    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-home\") pod \"e0fbc1ea-317d-4b98-966d-a3007461ded0\" (UID: \"e0fbc1ea-317d-4b98-966d-a3007461ded0\") "
Sep 16 19:51:51.036806 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0fbc1ea-317d-4b98-966d-a3007461ded0" (UID: "e0fbc1ea-317d-4b98-966d-a3007461ded0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.036806 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.036724    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fbc1ea-317d-4b98-966d-a3007461ded0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0fbc1ea-317d-4b98-966d-a3007461ded0" (UID: "e0fbc1ea-317d-4b98-966d-a3007461ded0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:51:51.037138 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.037063    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbc1ea-317d-4b98-966d-a3007461ded0-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.037138 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.037089    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.037365 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.037320    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-home" (OuterVolumeSpecName: "home") pod "e0fbc1ea-317d-4b98-966d-a3007461ded0" (UID: "e0fbc1ea-317d-4b98-966d-a3007461ded0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.039043 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.039020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-dshm" (OuterVolumeSpecName: "dshm") pod "e0fbc1ea-317d-4b98-966d-a3007461ded0" (UID: "e0fbc1ea-317d-4b98-966d-a3007461ded0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.039161 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.039119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0fbc1ea-317d-4b98-966d-a3007461ded0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0fbc1ea-317d-4b98-966d-a3007461ded0" (UID: "e0fbc1ea-317d-4b98-966d-a3007461ded0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:51:51.039783 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.039757    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fbc1ea-317d-4b98-966d-a3007461ded0-kube-api-access-skrzl" (OuterVolumeSpecName: "kube-api-access-skrzl") pod "e0fbc1ea-317d-4b98-966d-a3007461ded0" (UID: "e0fbc1ea-317d-4b98-966d-a3007461ded0"). InnerVolumeSpecName "kube-api-access-skrzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:51:51.049395 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.049363    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0fbc1ea-317d-4b98-966d-a3007461ded0" (UID: "e0fbc1ea-317d-4b98-966d-a3007461ded0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.062670 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.062639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0fbc1ea-317d-4b98-966d-a3007461ded0" (UID: "e0fbc1ea-317d-4b98-966d-a3007461ded0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:51.137605 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.137563    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skrzl\" (UniqueName: \"kubernetes.io/projected/e0fbc1ea-317d-4b98-966d-a3007461ded0-kube-api-access-skrzl\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.137605 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.137596    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fbc1ea-317d-4b98-966d-a3007461ded0-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.137605 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.137612    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.137856 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.137623    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.137856 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.137634    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.137856 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.137643    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fbc1ea-317d-4b98-966d-a3007461ded0-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:51:51.187341 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.187306    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerID="c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6" exitCode=0
Sep 16 19:51:51.187515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.187366    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"
Sep 16 19:51:51.187515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.187375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" event={"ID":"e0fbc1ea-317d-4b98-966d-a3007461ded0","Type":"ContainerDied","Data":"c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6"}
Sep 16 19:51:51.187515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.187447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j" event={"ID":"e0fbc1ea-317d-4b98-966d-a3007461ded0","Type":"ContainerDied","Data":"9ae1aaeb69cbc508e1a116df8a20636616a2b3e13374317bbce88a52a0f233fa"}
Sep 16 19:51:51.187515 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.187476    2580 scope.go:117] "RemoveContainer" containerID="c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6"
Sep 16 19:51:51.197287 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.197268    2580 scope.go:117] "RemoveContainer" containerID="47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e"
Sep 16 19:51:51.206114 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.206092    2580 scope.go:117] "RemoveContainer" containerID="c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6"
Sep 16 19:51:51.206397 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:51:51.206374    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6\": container with ID starting with c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6 not found: ID does not exist" containerID="c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6"
Sep 16 19:51:51.206471 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.206408    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6"} err="failed to get container status \"c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6\": rpc error: code = NotFound desc = could not find container \"c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6\": container with ID starting with c7cc9d285010eadc192d5ca3b07d8bf9127395bf258cb316ed53895e1efa11a6 not found: ID does not exist"
Sep 16 19:51:51.206471 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.206428    2580 scope.go:117] "RemoveContainer" containerID="47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e"
Sep 16 19:51:51.206692 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:51:51.206674    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e\": container with ID starting with 47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e not found: ID does not exist" containerID="47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e"
Sep 16 19:51:51.206738 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.206698    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e"} err="failed to get container status \"47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e\": rpc error: code = NotFound desc = could not find container \"47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e\": container with ID starting with 47aed78093f4bcfff3de197527e8ea5b44b99f554a2527dfa395ec624e082c8e not found: ID does not exist"
Sep 16 19:51:51.220113 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.220088    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"]
Sep 16 19:51:51.227277 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:51.227255    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79p776j"]
Sep 16 19:51:52.276309 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:51:52.276273    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" path="/var/lib/kubelet/pods/e0fbc1ea-317d-4b98-966d-a3007461ded0/volumes"
Sep 16 19:52:03.835720 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.835679    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"]
Sep 16 19:52:03.836131 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.836108    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="storage-initializer"
Sep 16 19:52:03.836131 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.836125    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="storage-initializer"
Sep 16 19:52:03.836206 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.836137    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1599806-fb1e-45d7-bf98-60cabec18df6" containerName="storage-initializer"
Sep 16 19:52:03.836206 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.836144    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1599806-fb1e-45d7-bf98-60cabec18df6" containerName="storage-initializer"
Sep 16 19:52:03.836206 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.836152    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main"
Sep 16 19:52:03.836206 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.836159    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main"
Sep 16 19:52:03.836336 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.836244    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0fbc1ea-317d-4b98-966d-a3007461ded0" containerName="main"
Sep 16 19:52:03.836336 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.836252    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1599806-fb1e-45d7-bf98-60cabec18df6" containerName="storage-initializer"
Sep 16 19:52:03.839283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.839260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:03.841597 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.841574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 16 19:52:03.842011 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.841994    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 19:52:03.842744 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.842722    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 19:52:03.842835 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.842738    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-czkfv\""
Sep 16 19:52:03.848675 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.848652    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"]
Sep 16 19:52:03.956747 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:03.956698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczpr\" (UniqueName: \"kubernetes.io/projected/34f4d0f8-ed92-47d4-88a3-023c25f90fa7-kube-api-access-rczpr\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:04.058287 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:04.058245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczpr\" (UniqueName: \"kubernetes.io/projected/34f4d0f8-ed92-47d4-88a3-023c25f90fa7-kube-api-access-rczpr\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:04.066646 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:04.066611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczpr\" (UniqueName: \"kubernetes.io/projected/34f4d0f8-ed92-47d4-88a3-023c25f90fa7-kube-api-access-rczpr\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:05.517905 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:05.517862    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"]
Sep 16 19:52:05.524215 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:05.524187    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-2mfgr"]
Sep 16 19:52:06.275453 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:06.275417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:06.275809 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:06.275787    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1599806-fb1e-45d7-bf98-60cabec18df6" path="/var/lib/kubelet/pods/f1599806-fb1e-45d7-bf98-60cabec18df6/volumes"
Sep 16 19:52:06.281393 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:06.281370    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:06.376856 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:06.376824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:06.377059 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:06.376952    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:06.380843 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:06.380821    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") DevicePath \"csi-4a6aa3d48e9cc03cdf6481991449bf765639a2d1b2ce72d717ac87cec14cff7e\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:06.548299 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:06.548263    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/dd931231270b48cb4cee8c64e7f689b8f1564e3b0455ef756768af2b40f3848e/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:06.563670 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:06.563645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"e2e-pvc-lora-download-w75gn\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:06.848559 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:06.848484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:07.126165 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:07.126136    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"]
Sep 16 19:52:07.128326 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:52:07.128298    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34f4d0f8_ed92_47d4_88a3_023c25f90fa7.slice/crio-9a4dd1c089d2fc4fb805c14fc194225b4f0ebf7864384ab03ee2dcc706d77902 WatchSource:0}: Error finding container 9a4dd1c089d2fc4fb805c14fc194225b4f0ebf7864384ab03ee2dcc706d77902: Status 404 returned error can't find the container with id 9a4dd1c089d2fc4fb805c14fc194225b4f0ebf7864384ab03ee2dcc706d77902
Sep 16 19:52:07.269335 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:07.269297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn" event={"ID":"34f4d0f8-ed92-47d4-88a3-023c25f90fa7","Type":"ContainerStarted","Data":"a9600cc067bcf8d3508c799204e6be920af4d2d74d755c4185a1eb59680f140f"}
Sep 16 19:52:07.269335 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:07.269341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn" event={"ID":"34f4d0f8-ed92-47d4-88a3-023c25f90fa7","Type":"ContainerStarted","Data":"9a4dd1c089d2fc4fb805c14fc194225b4f0ebf7864384ab03ee2dcc706d77902"}
Sep 16 19:52:07.297819 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:07.297763    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn" podStartSLOduration=7.297743641 podStartE2EDuration="7.297743641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:07.296685831 +0000 UTC m=+4107.558362843" watchObservedRunningTime="2026-09-16 19:52:07.297743641 +0000 UTC m=+4107.559420649"
Sep 16 19:52:09.283061 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:09.282970    2580 generic.go:356] "Generic (PLEG): container finished" podID="34f4d0f8-ed92-47d4-88a3-023c25f90fa7" containerID="a9600cc067bcf8d3508c799204e6be920af4d2d74d755c4185a1eb59680f140f" exitCode=0
Sep 16 19:52:09.283061 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:09.283030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn" event={"ID":"34f4d0f8-ed92-47d4-88a3-023c25f90fa7","Type":"ContainerDied","Data":"a9600cc067bcf8d3508c799204e6be920af4d2d74d755c4185a1eb59680f140f"}
Sep 16 19:52:10.417065 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:10.417035    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:10.510715 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:10.510679    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") pod \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") "
Sep 16 19:52:10.510915 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:10.510766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rczpr\" (UniqueName: \"kubernetes.io/projected/34f4d0f8-ed92-47d4-88a3-023c25f90fa7-kube-api-access-rczpr\") pod \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\" (UID: \"34f4d0f8-ed92-47d4-88a3-023c25f90fa7\") "
Sep 16 19:52:10.513098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:10.513065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f4d0f8-ed92-47d4-88a3-023c25f90fa7-kube-api-access-rczpr" (OuterVolumeSpecName: "kube-api-access-rczpr") pod "34f4d0f8-ed92-47d4-88a3-023c25f90fa7" (UID: "34f4d0f8-ed92-47d4-88a3-023c25f90fa7"). InnerVolumeSpecName "kube-api-access-rczpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:52:10.513439 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:10.513421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94" (OuterVolumeSpecName: "model-storage") pod "34f4d0f8-ed92-47d4-88a3-023c25f90fa7" (UID: "34f4d0f8-ed92-47d4-88a3-023c25f90fa7"). InnerVolumeSpecName "pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:52:10.612358 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:10.612271    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczpr\" (UniqueName: \"kubernetes.io/projected/34f4d0f8-ed92-47d4-88a3-023c25f90fa7-kube-api-access-rczpr\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:52:10.612358 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:10.612324    2580 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") on node \"ip-10-0-139-95.ec2.internal\" "
Sep 16 19:52:11.293610 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:11.293579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn" event={"ID":"34f4d0f8-ed92-47d4-88a3-023c25f90fa7","Type":"ContainerDied","Data":"9a4dd1c089d2fc4fb805c14fc194225b4f0ebf7864384ab03ee2dcc706d77902"}
Sep 16 19:52:11.293799 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:11.293619    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a4dd1c089d2fc4fb805c14fc194225b4f0ebf7864384ab03ee2dcc706d77902"
Sep 16 19:52:11.293799 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:11.293591    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"
Sep 16 19:52:11.582461 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:11.582442    2580 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94") on node "ip-10-0-139-95.ec2.internal"
Sep 16 19:52:11.620210 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:11.620172    2580 reconciler_common.go:299] "Volume detached for volume \"pvc-0aedfc2c-223d-4c6c-b480-93cbde3a0e1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0eb6e5be7fc1f4c94\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"csi-4a6aa3d48e9cc03cdf6481991449bf765639a2d1b2ce72d717ac87cec14cff7e\""
Sep 16 19:52:43.984070 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.984032    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"]
Sep 16 19:52:43.984592 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.984570    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34f4d0f8-ed92-47d4-88a3-023c25f90fa7" containerName="storage-initializer"
Sep 16 19:52:43.984670 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.984595    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34f4d0f8-ed92-47d4-88a3-023c25f90fa7" containerName="storage-initializer"
Sep 16 19:52:43.984730 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.984684    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34f4d0f8-ed92-47d4-88a3-023c25f90fa7" containerName="storage-initializer"
Sep 16 19:52:43.989556 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.989532    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:43.996863 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.996511    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:52:43.996863 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.996827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:52:43.996863 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.996845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:52:43.997177 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.997146    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-frw2j\""
Sep 16 19:52:43.997386 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.997367    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:52:43.997661 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.997480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cm5jg\""
Sep 16 19:52:43.997805 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:43.997791    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:52:44.006196 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.006172    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"]
Sep 16 19:52:44.087167 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.087133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh66x\" (UniqueName: \"kubernetes.io/projected/f62a466d-4367-477b-9518-f494beb3b6ec-kube-api-access-bh66x\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.087328 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.087173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.087328 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.087200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62a466d-4367-477b-9518-f494beb3b6ec-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.087328 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.087225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.087328 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.087241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f62a466d-4367-477b-9518-f494beb3b6ec-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.087328 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.087278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.087328 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.087301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-home\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.087533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.087369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-dshm\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.187721 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.187690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-dshm\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.187721 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.187723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh66x\" (UniqueName: \"kubernetes.io/projected/f62a466d-4367-477b-9518-f494beb3b6ec-kube-api-access-bh66x\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.187943 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.187745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.187943 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.187770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62a466d-4367-477b-9518-f494beb3b6ec-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.187943 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.187806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.187943 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.187838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f62a466d-4367-477b-9518-f494beb3b6ec-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.187943 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.187874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.187943 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.187932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-home\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.188261 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.188196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.188323 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.188297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.188323 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.188316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-home\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.188548 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.188527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.188917 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.188872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f62a466d-4367-477b-9518-f494beb3b6ec-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.190301 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.190281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-dshm\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.190499 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.190483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62a466d-4367-477b-9518-f494beb3b6ec-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.197671 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.197646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh66x\" (UniqueName: \"kubernetes.io/projected/f62a466d-4367-477b-9518-f494beb3b6ec-kube-api-access-bh66x\") pod \"custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.300156 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.300091    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:44.582796 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.582754    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"]
Sep 16 19:52:44.585178 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:52:44.585153    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62a466d_4367_477b_9518_f494beb3b6ec.slice/crio-c5ddf4ac083dd3813954b2eaa380f743e8e3d957876f56392666ca0615d7f785 WatchSource:0}: Error finding container c5ddf4ac083dd3813954b2eaa380f743e8e3d957876f56392666ca0615d7f785: Status 404 returned error can't find the container with id c5ddf4ac083dd3813954b2eaa380f743e8e3d957876f56392666ca0615d7f785
Sep 16 19:52:44.587305 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:44.587285    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:52:45.452674 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:45.452634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" event={"ID":"f62a466d-4367-477b-9518-f494beb3b6ec","Type":"ContainerStarted","Data":"c5ddf4ac083dd3813954b2eaa380f743e8e3d957876f56392666ca0615d7f785"}
Sep 16 19:52:46.459352 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:46.459317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" event={"ID":"f62a466d-4367-477b-9518-f494beb3b6ec","Type":"ContainerStarted","Data":"6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5"}
Sep 16 19:52:46.459736 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:46.459418    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:47.469820 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:47.469760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" event={"ID":"f62a466d-4367-477b-9518-f494beb3b6ec","Type":"ContainerStarted","Data":"6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa"}
Sep 16 19:52:49.482423 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:49.482387    2580 generic.go:356] "Generic (PLEG): container finished" podID="f62a466d-4367-477b-9518-f494beb3b6ec" containerID="6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa" exitCode=0
Sep 16 19:52:49.482853 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:49.482461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" event={"ID":"f62a466d-4367-477b-9518-f494beb3b6ec","Type":"ContainerDied","Data":"6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa"}
Sep 16 19:52:50.489744 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:50.489709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" event={"ID":"f62a466d-4367-477b-9518-f494beb3b6ec","Type":"ContainerStarted","Data":"f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279"}
Sep 16 19:52:50.517400 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:50.517350    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podStartSLOduration=3.99589383 podStartE2EDuration="7.517334371s" totalImagesPullingTime="1.373908454s" totalInitContainerRuntime="2.147532087s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:50.514118848 +0000 UTC m=+4150.775795873" watchObservedRunningTime="2026-09-16 19:52:50.517334371 +0000 UTC m=+4150.779011380"
Sep 16 19:52:54.300776 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:54.300737    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:54.301282 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:54.300811    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:52:54.302210 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:52:54.302184    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:53:04.301519 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:53:04.301478    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:53:04.313823 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:53:04.313794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:53:14.301093 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:53:14.301054    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:53:24.301064 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:53:24.301020    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:53:34.300604 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:53:34.300556    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:53:44.300882 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:53:44.300840    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:53:54.301234 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:53:54.301183    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:54:04.301163 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:54:04.301055    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:54:14.301453 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:54:14.301404    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:54:24.300944 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:54:24.300869    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:54:34.301529 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:54:34.301475    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8001/health\": dial tcp 10.134.0.71:8001: connect: connection refused"
Sep 16 19:54:44.310667 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:54:44.310623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:54:44.322826 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:54:44.322804    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:55:08.625656 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:08.625621    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"]
Sep 16 19:55:08.629473 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:08.629449    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-w75gn"]
Sep 16 19:55:10.276180 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:10.276142    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34f4d0f8-ed92-47d4-88a3-023c25f90fa7" path="/var/lib/kubelet/pods/34f4d0f8-ed92-47d4-88a3-023c25f90fa7/volumes"
Sep 16 19:55:32.012020 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:32.011938    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"]
Sep 16 19:55:40.825467 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:40.825436    2580 scope.go:117] "RemoveContainer" containerID="616dddba15b0985938a60a47452dea16138b682cd7af9f8d22625cf854550f13"
Sep 16 19:55:47.036249 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.036190    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main" containerID="cri-o://f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279" gracePeriod=45
Sep 16 19:55:47.581295 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.581245    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="llm-d-routing-sidecar" containerID="cri-o://6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5" gracePeriod=45
Sep 16 19:55:47.819985 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.819958    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:55:47.988474 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.988440    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f62a466d-4367-477b-9518-f494beb3b6ec-cabundle-cert\") pod \"f62a466d-4367-477b-9518-f494beb3b6ec\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") "
Sep 16 19:55:47.988645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.988480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-home\") pod \"f62a466d-4367-477b-9518-f494beb3b6ec\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") "
Sep 16 19:55:47.988645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.988529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-kserve-provision-location\") pod \"f62a466d-4367-477b-9518-f494beb3b6ec\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") "
Sep 16 19:55:47.988645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.988570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-dshm\") pod \"f62a466d-4367-477b-9518-f494beb3b6ec\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") "
Sep 16 19:55:47.988645 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.988611    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh66x\" (UniqueName: \"kubernetes.io/projected/f62a466d-4367-477b-9518-f494beb3b6ec-kube-api-access-bh66x\") pod \"f62a466d-4367-477b-9518-f494beb3b6ec\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") "
Sep 16 19:55:47.988863 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.988647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-tmp-dir\") pod \"f62a466d-4367-477b-9518-f494beb3b6ec\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") "
Sep 16 19:55:47.988863 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.988673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-model-cache\") pod \"f62a466d-4367-477b-9518-f494beb3b6ec\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") "
Sep 16 19:55:47.988863 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.988709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62a466d-4367-477b-9518-f494beb3b6ec-tls-certs\") pod \"f62a466d-4367-477b-9518-f494beb3b6ec\" (UID: \"f62a466d-4367-477b-9518-f494beb3b6ec\") "
Sep 16 19:55:47.988863 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.988802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62a466d-4367-477b-9518-f494beb3b6ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f62a466d-4367-477b-9518-f494beb3b6ec" (UID: "f62a466d-4367-477b-9518-f494beb3b6ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:55:47.989110 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.989024    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f62a466d-4367-477b-9518-f494beb3b6ec-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:55:47.989204 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.989172    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "f62a466d-4367-477b-9518-f494beb3b6ec" (UID: "f62a466d-4367-477b-9518-f494beb3b6ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:47.989283 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.989261    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-home" (OuterVolumeSpecName: "home") pod "f62a466d-4367-477b-9518-f494beb3b6ec" (UID: "f62a466d-4367-477b-9518-f494beb3b6ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:47.991235 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.991204    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62a466d-4367-477b-9518-f494beb3b6ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f62a466d-4367-477b-9518-f494beb3b6ec" (UID: "f62a466d-4367-477b-9518-f494beb3b6ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:55:47.991568 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.991549    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-dshm" (OuterVolumeSpecName: "dshm") pod "f62a466d-4367-477b-9518-f494beb3b6ec" (UID: "f62a466d-4367-477b-9518-f494beb3b6ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:47.991641 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:47.991585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62a466d-4367-477b-9518-f494beb3b6ec-kube-api-access-bh66x" (OuterVolumeSpecName: "kube-api-access-bh66x") pod "f62a466d-4367-477b-9518-f494beb3b6ec" (UID: "f62a466d-4367-477b-9518-f494beb3b6ec"). InnerVolumeSpecName "kube-api-access-bh66x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:55:48.005608 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.005573    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f62a466d-4367-477b-9518-f494beb3b6ec" (UID: "f62a466d-4367-477b-9518-f494beb3b6ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:48.014661 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.014641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f62a466d-4367-477b-9518-f494beb3b6ec" (UID: "f62a466d-4367-477b-9518-f494beb3b6ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:48.089858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.089785    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.089858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.089813    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.089858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.089828    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.089858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.089841    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh66x\" (UniqueName: \"kubernetes.io/projected/f62a466d-4367-477b-9518-f494beb3b6ec-kube-api-access-bh66x\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.089858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.089854    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.090317 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.089866    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62a466d-4367-477b-9518-f494beb3b6ec-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.090317 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.089877    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62a466d-4367-477b-9518-f494beb3b6ec-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:55:48.376358 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.376281    2580 generic.go:356] "Generic (PLEG): container finished" podID="f62a466d-4367-477b-9518-f494beb3b6ec" containerID="f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279" exitCode=0
Sep 16 19:55:48.376358 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.376311    2580 generic.go:356] "Generic (PLEG): container finished" podID="f62a466d-4367-477b-9518-f494beb3b6ec" containerID="6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5" exitCode=0
Sep 16 19:55:48.376358 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.376336    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"
Sep 16 19:55:48.376578 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.376373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" event={"ID":"f62a466d-4367-477b-9518-f494beb3b6ec","Type":"ContainerDied","Data":"f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279"}
Sep 16 19:55:48.376578 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.376425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" event={"ID":"f62a466d-4367-477b-9518-f494beb3b6ec","Type":"ContainerDied","Data":"6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5"}
Sep 16 19:55:48.376578 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.376444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg" event={"ID":"f62a466d-4367-477b-9518-f494beb3b6ec","Type":"ContainerDied","Data":"c5ddf4ac083dd3813954b2eaa380f743e8e3d957876f56392666ca0615d7f785"}
Sep 16 19:55:48.376578 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.376463    2580 scope.go:117] "RemoveContainer" containerID="f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279"
Sep 16 19:55:48.385262 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.385239    2580 scope.go:117] "RemoveContainer" containerID="6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa"
Sep 16 19:55:48.393047 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.393029    2580 scope.go:117] "RemoveContainer" containerID="6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5"
Sep 16 19:55:48.398307 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.398280    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"]
Sep 16 19:55:48.401062 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.401035    2580 scope.go:117] "RemoveContainer" containerID="f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279"
Sep 16 19:55:48.401337 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:55:48.401317    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279\": container with ID starting with f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279 not found: ID does not exist" containerID="f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279"
Sep 16 19:55:48.401391 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.401344    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279"} err="failed to get container status \"f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279\": rpc error: code = NotFound desc = could not find container \"f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279\": container with ID starting with f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279 not found: ID does not exist"
Sep 16 19:55:48.401391 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.401370    2580 scope.go:117] "RemoveContainer" containerID="6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa"
Sep 16 19:55:48.401626 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:55:48.401607    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa\": container with ID starting with 6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa not found: ID does not exist" containerID="6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa"
Sep 16 19:55:48.401671 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.401629    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa"} err="failed to get container status \"6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa\": rpc error: code = NotFound desc = could not find container \"6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa\": container with ID starting with 6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa not found: ID does not exist"
Sep 16 19:55:48.401671 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.401643    2580 scope.go:117] "RemoveContainer" containerID="6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5"
Sep 16 19:55:48.401955 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:55:48.401883    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5\": container with ID starting with 6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5 not found: ID does not exist" containerID="6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5"
Sep 16 19:55:48.402049 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.401962    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5"} err="failed to get container status \"6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5\": rpc error: code = NotFound desc = could not find container \"6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5\": container with ID starting with 6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5 not found: ID does not exist"
Sep 16 19:55:48.402049 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.401979    2580 scope.go:117] "RemoveContainer" containerID="f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279"
Sep 16 19:55:48.402271 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.402254    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279"} err="failed to get container status \"f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279\": rpc error: code = NotFound desc = could not find container \"f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279\": container with ID starting with f18e50b0a16ffc8daf29adf1ad26bc67444539d596c94ec2ef5ec82c79d39279 not found: ID does not exist"
Sep 16 19:55:48.402320 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.402272    2580 scope.go:117] "RemoveContainer" containerID="6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa"
Sep 16 19:55:48.402550 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.402534    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa"} err="failed to get container status \"6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa\": rpc error: code = NotFound desc = could not find container \"6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa\": container with ID starting with 6f5d17e903b4234759dae54e034a8fa61192fb247e645f84053f36073bcc0dfa not found: ID does not exist"
Sep 16 19:55:48.402618 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.402550    2580 scope.go:117] "RemoveContainer" containerID="6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5"
Sep 16 19:55:48.402805 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.402783    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5"} err="failed to get container status \"6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5\": rpc error: code = NotFound desc = could not find container \"6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5\": container with ID starting with 6381279b2f901b126bcea5713e9af49c93d91d88a4cf24749e87e912bbb23ba5 not found: ID does not exist"
Sep 16 19:55:48.406439 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:48.406418    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5dc8b6b86c-7s7bg"]
Sep 16 19:55:50.276390 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:55:50.276348    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" path="/var/lib/kubelet/pods/f62a466d-4367-477b-9518-f494beb3b6ec/volumes"
Sep 16 19:56:22.568696 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.568661    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"]
Sep 16 19:56:22.571384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.569287    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="storage-initializer"
Sep 16 19:56:22.571384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.569307    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="storage-initializer"
Sep 16 19:56:22.571384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.569324    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="llm-d-routing-sidecar"
Sep 16 19:56:22.571384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.569333    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="llm-d-routing-sidecar"
Sep 16 19:56:22.571384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.569367    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main"
Sep 16 19:56:22.571384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.569377    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main"
Sep 16 19:56:22.571384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.569469    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="llm-d-routing-sidecar"
Sep 16 19:56:22.571384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.569484    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62a466d-4367-477b-9518-f494beb3b6ec" containerName="main"
Sep 16 19:56:22.572858 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.572839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.577342 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.577320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-pmf5b\""
Sep 16 19:56:22.577849 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.577831    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:56:22.580378 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.580337    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:56:22.580974 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.580561    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:56:22.593839 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.593815    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"]
Sep 16 19:56:22.669146 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.669102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.669146 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.669150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-podinfo\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.669399 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.669183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc2z7\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-kube-api-access-fc2z7\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.669399 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.669247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-token\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.669399 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.669294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-data\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.669399 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.669321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d535f54-ea0a-4a70-b066-925933b03a4e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.669399 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.669379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-certs\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.669610 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.669401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-envoy\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.669610 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.669444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-credential-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.770754 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.770715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-token\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771000 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.770769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-data\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771000 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.770789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d535f54-ea0a-4a70-b066-925933b03a4e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771000 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.770830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-certs\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771000 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.770847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-envoy\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771000 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.770878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-credential-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771290 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.770996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771290 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.771031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-podinfo\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771290 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.771173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc2z7\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-kube-api-access-fc2z7\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771290 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.771221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-certs\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771290 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.771249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-data\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771479 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.771380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-credential-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.771533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.771491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.772020 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.771999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d535f54-ea0a-4a70-b066-925933b03a4e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.773413 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.773396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-envoy\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.773716 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.773697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-podinfo\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.801679 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.801648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-token\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.802817 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.802793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc2z7\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-kube-api-access-fc2z7\") pod \"router-gateway-2-openshift-default-5858dff66d-jg8mh\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:22.884790 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:22.884706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:23.183525 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:23.183491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"]
Sep 16 19:56:23.187268 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:56:23.187244    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d535f54_ea0a_4a70_b066_925933b03a4e.slice/crio-a95cd3b9ad5e520fe77aff5cff05e3cfee6afcd6ce537be33a2ce3131a42a9d9 WatchSource:0}: Error finding container a95cd3b9ad5e520fe77aff5cff05e3cfee6afcd6ce537be33a2ce3131a42a9d9: Status 404 returned error can't find the container with id a95cd3b9ad5e520fe77aff5cff05e3cfee6afcd6ce537be33a2ce3131a42a9d9
Sep 16 19:56:23.189707 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:23.189673    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:56:23.189822 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:23.189739    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:56:23.189822 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:23.189770    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:56:23.543440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:23.543350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" event={"ID":"5d535f54-ea0a-4a70-b066-925933b03a4e","Type":"ContainerStarted","Data":"a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d"}
Sep 16 19:56:23.543440 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:23.543403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" event={"ID":"5d535f54-ea0a-4a70-b066-925933b03a4e","Type":"ContainerStarted","Data":"a95cd3b9ad5e520fe77aff5cff05e3cfee6afcd6ce537be33a2ce3131a42a9d9"}
Sep 16 19:56:23.567984 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:23.567931    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" podStartSLOduration=1.567914065 podStartE2EDuration="1.567914065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:56:23.566557452 +0000 UTC m=+4363.828234470" watchObservedRunningTime="2026-09-16 19:56:23.567914065 +0000 UTC m=+4363.829591070"
Sep 16 19:56:23.885938 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:23.885816    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:24.890638 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:24.890606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:25.552182 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:25.552154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:25.553099 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:25.553078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:35.611047 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:35.610945    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerID="a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d" exitCode=0
Sep 16 19:56:35.611047 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:35.611031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" event={"ID":"5d535f54-ea0a-4a70-b066-925933b03a4e","Type":"ContainerDied","Data":"a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d"}
Sep 16 19:56:35.611555 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:35.611430    2580 scope.go:117] "RemoveContainer" containerID="a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d"
Sep 16 19:56:35.612402 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:35.612376    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:56:35.612468 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:35.612448    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:56:35.612505 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:35.612478    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:56:36.618378 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:36.618275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" event={"ID":"5d535f54-ea0a-4a70-b066-925933b03a4e","Type":"ContainerStarted","Data":"04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90"}
Sep 16 19:56:36.885249 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:36.885127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:36.886367 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:36.886327    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 16 19:56:37.885480 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:37.885434    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 16 19:56:38.885544 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:38.885493    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 16 19:56:39.885911 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:39.885847    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 16 19:56:40.552384 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:40.552343    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:40.885241 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:40.885205    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 16 19:56:41.891015 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:41.890976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:41.892088 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:41.892064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:56:42.650959 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.650928    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"]
Sep 16 19:56:42.654628 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.654605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.657747 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.657725    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:56:42.657859 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.657778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-xsksx\""
Sep 16 19:56:42.658021 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.658006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4m4fc\""
Sep 16 19:56:42.658076 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.658022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:56:42.658127 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.658025    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:56:42.681306 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.681278    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"]
Sep 16 19:56:42.776597 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.776563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-model-cache\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.776597 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.776593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b13951-9161-4100-bcb3-26a7eebe6b17-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.776794 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.776617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpgn\" (UniqueName: \"kubernetes.io/projected/37b13951-9161-4100-bcb3-26a7eebe6b17-kube-api-access-5rpgn\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.776794 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.776771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-home\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.776864 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.776825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-dshm\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.776864 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.776856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.777003 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.776971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b13951-9161-4100-bcb3-26a7eebe6b17-tls-certs\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.777100 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.777081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878312 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-home\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878482 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-dshm\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878482 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878482 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b13951-9161-4100-bcb3-26a7eebe6b17-tls-certs\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878482 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878705 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-model-cache\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878705 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b13951-9161-4100-bcb3-26a7eebe6b17-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878705 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpgn\" (UniqueName: \"kubernetes.io/projected/37b13951-9161-4100-bcb3-26a7eebe6b17-kube-api-access-5rpgn\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878705 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-home\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878937 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.878937 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.879037 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.878982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-model-cache\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.879264 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.879236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b13951-9161-4100-bcb3-26a7eebe6b17-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.880779 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.880759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-dshm\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.881347 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.881325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b13951-9161-4100-bcb3-26a7eebe6b17-tls-certs\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.895097 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.895071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpgn\" (UniqueName: \"kubernetes.io/projected/37b13951-9161-4100-bcb3-26a7eebe6b17-kube-api-access-5rpgn\") pod \"router-with-refs-pd-test-kserve-7f7f7df49c-jmphq\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:42.964828 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:42.964801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:43.243644 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:43.243566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"]
Sep 16 19:56:43.246629 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:56:43.246594    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b13951_9161_4100_bcb3_26a7eebe6b17.slice/crio-ea11fa515396ba54cdd3cc8fb52f8e80e0e3a6b7200516025a86f20deb30566b WatchSource:0}: Error finding container ea11fa515396ba54cdd3cc8fb52f8e80e0e3a6b7200516025a86f20deb30566b: Status 404 returned error can't find the container with id ea11fa515396ba54cdd3cc8fb52f8e80e0e3a6b7200516025a86f20deb30566b
Sep 16 19:56:43.652142 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:43.652043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" event={"ID":"37b13951-9161-4100-bcb3-26a7eebe6b17","Type":"ContainerStarted","Data":"858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02"}
Sep 16 19:56:43.652142 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:43.652115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" event={"ID":"37b13951-9161-4100-bcb3-26a7eebe6b17","Type":"ContainerStarted","Data":"ea11fa515396ba54cdd3cc8fb52f8e80e0e3a6b7200516025a86f20deb30566b"}
Sep 16 19:56:43.652343 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:43.652211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:44.660498 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:44.660454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" event={"ID":"37b13951-9161-4100-bcb3-26a7eebe6b17","Type":"ContainerStarted","Data":"d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a"}
Sep 16 19:56:46.673711 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:46.673679    2580 generic.go:356] "Generic (PLEG): container finished" podID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerID="d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a" exitCode=0
Sep 16 19:56:46.674156 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:46.673722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" event={"ID":"37b13951-9161-4100-bcb3-26a7eebe6b17","Type":"ContainerDied","Data":"d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a"}
Sep 16 19:56:47.680962 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:47.680924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" event={"ID":"37b13951-9161-4100-bcb3-26a7eebe6b17","Type":"ContainerStarted","Data":"dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d"}
Sep 16 19:56:47.706698 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:47.706641    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podStartSLOduration=3.394934841 podStartE2EDuration="5.706623562s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.311688721s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:56:47.703623966 +0000 UTC m=+4387.965300977" watchObservedRunningTime="2026-09-16 19:56:47.706623562 +0000 UTC m=+4387.968300570"
Sep 16 19:56:52.965597 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:52.965553    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:52.966138 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:52.965635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:56:52.966731 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:56:52.966704    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:57:02.965373 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:57:02.965279    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:57:02.984212 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:57:02.984190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:57:12.965230 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:57:12.965177    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:57:22.966177 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:57:22.966132    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:57:32.965150 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:57:32.965104    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:57:42.965783 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:57:42.965725    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:57:52.965872 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:57:52.965826    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:58:02.965699 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:58:02.965644    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:58:12.965562 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:58:12.965514    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:58:22.965709 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:58:22.965656    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:58:32.965286 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:58:32.965175    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 16 19:58:40.871591 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:58:40.871551    2580 scope.go:117] "RemoveContainer" containerID="a9600cc067bcf8d3508c799204e6be920af4d2d74d755c4185a1eb59680f140f"
Sep 16 19:58:42.976526 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:58:42.976492    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:58:42.991830 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:58:42.991806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:58:52.858626 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:58:52.858584    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"]
Sep 16 19:59:07.882019 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:07.881975    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main" containerID="cri-o://dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d" gracePeriod=45
Sep 16 19:59:08.499819 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.499759    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="llm-d-routing-sidecar" containerID="cri-o://858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02" gracePeriod=45
Sep 16 19:59:08.740544 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.740517    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:59:08.821081 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-kserve-provision-location\") pod \"37b13951-9161-4100-bcb3-26a7eebe6b17\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") "
Sep 16 19:59:08.821269 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821098    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b13951-9161-4100-bcb3-26a7eebe6b17-cabundle-cert\") pod \"37b13951-9161-4100-bcb3-26a7eebe6b17\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") "
Sep 16 19:59:08.821269 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-model-cache\") pod \"37b13951-9161-4100-bcb3-26a7eebe6b17\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") "
Sep 16 19:59:08.821269 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-home\") pod \"37b13951-9161-4100-bcb3-26a7eebe6b17\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") "
Sep 16 19:59:08.821269 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b13951-9161-4100-bcb3-26a7eebe6b17-tls-certs\") pod \"37b13951-9161-4100-bcb3-26a7eebe6b17\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") "
Sep 16 19:59:08.821269 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-dshm\") pod \"37b13951-9161-4100-bcb3-26a7eebe6b17\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") "
Sep 16 19:59:08.821538 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rpgn\" (UniqueName: \"kubernetes.io/projected/37b13951-9161-4100-bcb3-26a7eebe6b17-kube-api-access-5rpgn\") pod \"37b13951-9161-4100-bcb3-26a7eebe6b17\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") "
Sep 16 19:59:08.821538 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-tmp-dir\") pod \"37b13951-9161-4100-bcb3-26a7eebe6b17\" (UID: \"37b13951-9161-4100-bcb3-26a7eebe6b17\") "
Sep 16 19:59:08.821538 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-model-cache" (OuterVolumeSpecName: "model-cache") pod "37b13951-9161-4100-bcb3-26a7eebe6b17" (UID: "37b13951-9161-4100-bcb3-26a7eebe6b17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.821538 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b13951-9161-4100-bcb3-26a7eebe6b17-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37b13951-9161-4100-bcb3-26a7eebe6b17" (UID: "37b13951-9161-4100-bcb3-26a7eebe6b17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:59:08.821747 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821617    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b13951-9161-4100-bcb3-26a7eebe6b17-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.821747 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821636    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.821877 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.821848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-home" (OuterVolumeSpecName: "home") pod "37b13951-9161-4100-bcb3-26a7eebe6b17" (UID: "37b13951-9161-4100-bcb3-26a7eebe6b17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.824098 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.824074    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b13951-9161-4100-bcb3-26a7eebe6b17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37b13951-9161-4100-bcb3-26a7eebe6b17" (UID: "37b13951-9161-4100-bcb3-26a7eebe6b17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:59:08.824255 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.824226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-dshm" (OuterVolumeSpecName: "dshm") pod "37b13951-9161-4100-bcb3-26a7eebe6b17" (UID: "37b13951-9161-4100-bcb3-26a7eebe6b17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.824255 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.824240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b13951-9161-4100-bcb3-26a7eebe6b17-kube-api-access-5rpgn" (OuterVolumeSpecName: "kube-api-access-5rpgn") pod "37b13951-9161-4100-bcb3-26a7eebe6b17" (UID: "37b13951-9161-4100-bcb3-26a7eebe6b17"). InnerVolumeSpecName "kube-api-access-5rpgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:59:08.838477 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.838450    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37b13951-9161-4100-bcb3-26a7eebe6b17" (UID: "37b13951-9161-4100-bcb3-26a7eebe6b17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.846416 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.846392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37b13951-9161-4100-bcb3-26a7eebe6b17" (UID: "37b13951-9161-4100-bcb3-26a7eebe6b17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:08.923074 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.923052    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.923074 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.923073    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b13951-9161-4100-bcb3-26a7eebe6b17-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.923394 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.923083    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.923394 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.923091    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rpgn\" (UniqueName: \"kubernetes.io/projected/37b13951-9161-4100-bcb3-26a7eebe6b17-kube-api-access-5rpgn\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.923394 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.923119    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:08.923394 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:08.923128    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b13951-9161-4100-bcb3-26a7eebe6b17-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:09.404775 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.404679    2580 generic.go:356] "Generic (PLEG): container finished" podID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerID="dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d" exitCode=0
Sep 16 19:59:09.404775 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.404714    2580 generic.go:356] "Generic (PLEG): container finished" podID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerID="858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02" exitCode=0
Sep 16 19:59:09.404775 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.404750    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"
Sep 16 19:59:09.405079 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.404767    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" event={"ID":"37b13951-9161-4100-bcb3-26a7eebe6b17","Type":"ContainerDied","Data":"dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d"}
Sep 16 19:59:09.405079 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.404840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" event={"ID":"37b13951-9161-4100-bcb3-26a7eebe6b17","Type":"ContainerDied","Data":"858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02"}
Sep 16 19:59:09.405079 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.404877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq" event={"ID":"37b13951-9161-4100-bcb3-26a7eebe6b17","Type":"ContainerDied","Data":"ea11fa515396ba54cdd3cc8fb52f8e80e0e3a6b7200516025a86f20deb30566b"}
Sep 16 19:59:09.405079 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.404912    2580 scope.go:117] "RemoveContainer" containerID="dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d"
Sep 16 19:59:09.414253 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.414237    2580 scope.go:117] "RemoveContainer" containerID="d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a"
Sep 16 19:59:09.423028 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.423011    2580 scope.go:117] "RemoveContainer" containerID="858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02"
Sep 16 19:59:09.430524 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.430507    2580 scope.go:117] "RemoveContainer" containerID="dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d"
Sep 16 19:59:09.430797 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:59:09.430778    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d\": container with ID starting with dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d not found: ID does not exist" containerID="dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d"
Sep 16 19:59:09.430870 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.430802    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d"} err="failed to get container status \"dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d\": rpc error: code = NotFound desc = could not find container \"dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d\": container with ID starting with dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d not found: ID does not exist"
Sep 16 19:59:09.430870 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.430817    2580 scope.go:117] "RemoveContainer" containerID="d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a"
Sep 16 19:59:09.431200 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:59:09.431179    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a\": container with ID starting with d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a not found: ID does not exist" containerID="d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a"
Sep 16 19:59:09.431258 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.431205    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a"} err="failed to get container status \"d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a\": rpc error: code = NotFound desc = could not find container \"d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a\": container with ID starting with d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a not found: ID does not exist"
Sep 16 19:59:09.431258 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.431220    2580 scope.go:117] "RemoveContainer" containerID="858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02"
Sep 16 19:59:09.431471 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:59:09.431456    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02\": container with ID starting with 858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02 not found: ID does not exist" containerID="858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02"
Sep 16 19:59:09.431533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.431474    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02"} err="failed to get container status \"858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02\": rpc error: code = NotFound desc = could not find container \"858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02\": container with ID starting with 858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02 not found: ID does not exist"
Sep 16 19:59:09.431533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.431485    2580 scope.go:117] "RemoveContainer" containerID="dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d"
Sep 16 19:59:09.431694 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.431678    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d"} err="failed to get container status \"dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d\": rpc error: code = NotFound desc = could not find container \"dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d\": container with ID starting with dbe77616a1c03a3a995ed6b038ecb0c2a265a336f56017616c9be49275ba7e7d not found: ID does not exist"
Sep 16 19:59:09.431739 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.431694    2580 scope.go:117] "RemoveContainer" containerID="d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a"
Sep 16 19:59:09.431950 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.431930    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a"} err="failed to get container status \"d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a\": rpc error: code = NotFound desc = could not find container \"d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a\": container with ID starting with d08c7cea70304874bcb803f7f70de9ff63ec444cc0bb2db29cd5618207d8ba6a not found: ID does not exist"
Sep 16 19:59:09.432005 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.431952    2580 scope.go:117] "RemoveContainer" containerID="858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02"
Sep 16 19:59:09.432168 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.432147    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02"} err="failed to get container status \"858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02\": rpc error: code = NotFound desc = could not find container \"858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02\": container with ID starting with 858b5a2fed8fdb4eee5475c2bd06609666b94eb58e43e2102c4be4a5b9b8be02 not found: ID does not exist"
Sep 16 19:59:09.435259 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.435222    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"]
Sep 16 19:59:09.443060 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:09.443038    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7f7f7df49c-jmphq"]
Sep 16 19:59:10.275676 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:10.275642    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" path="/var/lib/kubelet/pods/37b13951-9161-4100-bcb3-26a7eebe6b17/volumes"
Sep 16 19:59:46.392712 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.392676    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"]
Sep 16 19:59:46.393152 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.393135    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="llm-d-routing-sidecar"
Sep 16 19:59:46.393202 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.393155    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="llm-d-routing-sidecar"
Sep 16 19:59:46.393202 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.393167    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="storage-initializer"
Sep 16 19:59:46.393202 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.393173    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="storage-initializer"
Sep 16 19:59:46.393202 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.393185    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main"
Sep 16 19:59:46.393202 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.393194    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main"
Sep 16 19:59:46.393373 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.393276    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="main"
Sep 16 19:59:46.393373 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.393285    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b13951-9161-4100-bcb3-26a7eebe6b17" containerName="llm-d-routing-sidecar"
Sep 16 19:59:46.397680 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.397653    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.400231 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.400199    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 19:59:46.400231 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.400223    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:59:46.400478 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.400297    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 19:59:46.400478 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.400311    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 19:59:46.400478 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.400324    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-82klq\""
Sep 16 19:59:46.400601 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.400523    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 19:59:46.408686 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.408653    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"]
Sep 16 19:59:46.449476 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.449434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4bgx\" (UniqueName: \"kubernetes.io/projected/a567a571-b2e9-41f0-82bc-d8d287474c49-kube-api-access-f4bgx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.449668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.449497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a571-b2e9-41f0-82bc-d8d287474c49-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.449668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.449531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.449668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.449547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.449668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.449578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.449668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.449604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.449668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.449627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a567a571-b2e9-41f0-82bc-d8d287474c49-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.449668 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.449655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551051 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.550972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551051 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551051 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a567a571-b2e9-41f0-82bc-d8d287474c49-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551475 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551475 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4bgx\" (UniqueName: \"kubernetes.io/projected/a567a571-b2e9-41f0-82bc-d8d287474c49-kube-api-access-f4bgx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551475 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a571-b2e9-41f0-82bc-d8d287474c49-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551475 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551475 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551787 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551787 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551787 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.551787 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.551774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.552405 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.552378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a571-b2e9-41f0-82bc-d8d287474c49-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.553942 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.553882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.554063 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.554050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a567a571-b2e9-41f0-82bc-d8d287474c49-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.559513 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.559486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4bgx\" (UniqueName: \"kubernetes.io/projected/a567a571-b2e9-41f0-82bc-d8d287474c49-kube-api-access-f4bgx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.710740 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.710695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:46.992823 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:46.992515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"]
Sep 16 19:59:46.995003 ip-10-0-139-95 kubenswrapper[2580]: W0916 19:59:46.994963    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda567a571_b2e9_41f0_82bc_d8d287474c49.slice/crio-6aa365e087ec53894faa40dd083a3ba93f5b4dc2f1e9f21866b4ad65647a0874 WatchSource:0}: Error finding container 6aa365e087ec53894faa40dd083a3ba93f5b4dc2f1e9f21866b4ad65647a0874: Status 404 returned error can't find the container with id 6aa365e087ec53894faa40dd083a3ba93f5b4dc2f1e9f21866b4ad65647a0874
Sep 16 19:59:47.531748 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:47.531707    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"]
Sep 16 19:59:47.532269 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:47.532014    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy" containerID="cri-o://04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90" gracePeriod=30
Sep 16 19:59:47.585989 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:47.585948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" event={"ID":"a567a571-b2e9-41f0-82bc-d8d287474c49","Type":"ContainerStarted","Data":"a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3"}
Sep 16 19:59:47.586204 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:47.586008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" event={"ID":"a567a571-b2e9-41f0-82bc-d8d287474c49","Type":"ContainerStarted","Data":"6aa365e087ec53894faa40dd083a3ba93f5b4dc2f1e9f21866b4ad65647a0874"}
Sep 16 19:59:49.597367 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:49.597332    2580 generic.go:356] "Generic (PLEG): container finished" podID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerID="a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3" exitCode=0
Sep 16 19:59:49.597747 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:49.597403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" event={"ID":"a567a571-b2e9-41f0-82bc-d8d287474c49","Type":"ContainerDied","Data":"a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3"}
Sep 16 19:59:50.605502 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:50.605461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" event={"ID":"a567a571-b2e9-41f0-82bc-d8d287474c49","Type":"ContainerStarted","Data":"f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7"}
Sep 16 19:59:50.630058 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:50.630010    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podStartSLOduration=2.250818235 podStartE2EDuration="4.62999549s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.379177255s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:59:50.627574889 +0000 UTC m=+4570.889251933" watchObservedRunningTime="2026-09-16 19:59:50.62999549 +0000 UTC m=+4570.891672501"
Sep 16 19:59:50.710831 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:50.710797    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:52.783992 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.783967    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:59:52.819177 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-podinfo\") pod \"5d535f54-ea0a-4a70-b066-925933b03a4e\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") "
Sep 16 19:59:52.819177 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-socket\") pod \"5d535f54-ea0a-4a70-b066-925933b03a4e\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") "
Sep 16 19:59:52.819392 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819227    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-data\") pod \"5d535f54-ea0a-4a70-b066-925933b03a4e\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") "
Sep 16 19:59:52.819392 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-credential-socket\") pod \"5d535f54-ea0a-4a70-b066-925933b03a4e\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") "
Sep 16 19:59:52.819392 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-token\") pod \"5d535f54-ea0a-4a70-b066-925933b03a4e\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") "
Sep 16 19:59:52.819392 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819340    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-envoy\") pod \"5d535f54-ea0a-4a70-b066-925933b03a4e\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") "
Sep 16 19:59:52.819392 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-certs\") pod \"5d535f54-ea0a-4a70-b066-925933b03a4e\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") "
Sep 16 19:59:52.819680 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819419    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc2z7\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-kube-api-access-fc2z7\") pod \"5d535f54-ea0a-4a70-b066-925933b03a4e\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") "
Sep 16 19:59:52.819680 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d535f54-ea0a-4a70-b066-925933b03a4e-istiod-ca-cert\") pod \"5d535f54-ea0a-4a70-b066-925933b03a4e\" (UID: \"5d535f54-ea0a-4a70-b066-925933b03a4e\") "
Sep 16 19:59:52.819680 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d535f54-ea0a-4a70-b066-925933b03a4e" (UID: "5d535f54-ea0a-4a70-b066-925933b03a4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:52.819841 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d535f54-ea0a-4a70-b066-925933b03a4e" (UID: "5d535f54-ea0a-4a70-b066-925933b03a4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:52.819841 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819776    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:52.820004 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.819976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d535f54-ea0a-4a70-b066-925933b03a4e" (UID: "5d535f54-ea0a-4a70-b066-925933b03a4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:52.820191 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.820168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d535f54-ea0a-4a70-b066-925933b03a4e" (UID: "5d535f54-ea0a-4a70-b066-925933b03a4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:52.820495 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.820470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d535f54-ea0a-4a70-b066-925933b03a4e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d535f54-ea0a-4a70-b066-925933b03a4e" (UID: "5d535f54-ea0a-4a70-b066-925933b03a4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:59:52.822691 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.822654    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d535f54-ea0a-4a70-b066-925933b03a4e" (UID: "5d535f54-ea0a-4a70-b066-925933b03a4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:59:52.822818 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.822789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-kube-api-access-fc2z7" (OuterVolumeSpecName: "kube-api-access-fc2z7") pod "5d535f54-ea0a-4a70-b066-925933b03a4e" (UID: "5d535f54-ea0a-4a70-b066-925933b03a4e"). InnerVolumeSpecName "kube-api-access-fc2z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:59:52.823177 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.823154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d535f54-ea0a-4a70-b066-925933b03a4e" (UID: "5d535f54-ea0a-4a70-b066-925933b03a4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:52.823533 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.823511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d535f54-ea0a-4a70-b066-925933b03a4e" (UID: "5d535f54-ea0a-4a70-b066-925933b03a4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:59:52.920246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.920202    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-data\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:52.920246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.920232    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-credential-socket\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:52.920246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.920241    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-token\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:52.920246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.920250    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-envoy\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:52.920246 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.920259    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d535f54-ea0a-4a70-b066-925933b03a4e-workload-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:52.920653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.920269    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc2z7\" (UniqueName: \"kubernetes.io/projected/5d535f54-ea0a-4a70-b066-925933b03a4e-kube-api-access-fc2z7\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:52.920653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.920278    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d535f54-ea0a-4a70-b066-925933b03a4e-istiod-ca-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:52.920653 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:52.920285    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d535f54-ea0a-4a70-b066-925933b03a4e-istio-podinfo\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 19:59:53.621601 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.621568    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerID="04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90" exitCode=0
Sep 16 19:59:53.621789 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.621614    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"
Sep 16 19:59:53.621789 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.621655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" event={"ID":"5d535f54-ea0a-4a70-b066-925933b03a4e","Type":"ContainerDied","Data":"04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90"}
Sep 16 19:59:53.621789 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.621712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh" event={"ID":"5d535f54-ea0a-4a70-b066-925933b03a4e","Type":"ContainerDied","Data":"a95cd3b9ad5e520fe77aff5cff05e3cfee6afcd6ce537be33a2ce3131a42a9d9"}
Sep 16 19:59:53.621789 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.621737    2580 scope.go:117] "RemoveContainer" containerID="04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90"
Sep 16 19:59:53.635090 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.631153    2580 scope.go:117] "RemoveContainer" containerID="a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d"
Sep 16 19:59:53.645518 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.645495    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"]
Sep 16 19:59:53.646642 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.646622    2580 scope.go:117] "RemoveContainer" containerID="04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90"
Sep 16 19:59:53.647029 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:59:53.646984    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90\": container with ID starting with 04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90 not found: ID does not exist" containerID="04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90"
Sep 16 19:59:53.647199 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.647026    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90"} err="failed to get container status \"04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90\": rpc error: code = NotFound desc = could not find container \"04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90\": container with ID starting with 04b92b178597735a78e903702d55ddbac98add85abfab4feb2773af2a85f4a90 not found: ID does not exist"
Sep 16 19:59:53.647199 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.647065    2580 scope.go:117] "RemoveContainer" containerID="a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d"
Sep 16 19:59:53.647434 ip-10-0-139-95 kubenswrapper[2580]: E0916 19:59:53.647387    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d\": container with ID starting with a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d not found: ID does not exist" containerID="a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d"
Sep 16 19:59:53.647541 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.647430    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d"} err="failed to get container status \"a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d\": rpc error: code = NotFound desc = could not find container \"a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d\": container with ID starting with a97106af3c7b01189bc3114ebec8beae287c3bfc20d67a2a655daea3325ae45d not found: ID does not exist"
Sep 16 19:59:53.650083 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:53.650059    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-jg8mh"]
Sep 16 19:59:54.276573 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:54.276531    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" path="/var/lib/kubelet/pods/5d535f54-ea0a-4a70-b066-925933b03a4e/volumes"
Sep 16 19:59:56.711690 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:56.711642    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 19:59:56.713292 ip-10-0-139-95 kubenswrapper[2580]: I0916 19:59:56.713263    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:00:06.711560 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:00:06.711472    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:00:16.711575 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:00:16.711528    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:00:26.711347 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:00:26.711301    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:00:36.711936 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:00:36.711856    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:00:46.711431 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:00:46.711382    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:00:56.711619 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:00:56.711574    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:01:06.711905 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:01:06.711840    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:01:16.711144 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:01:16.711102    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:01:26.712108 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:01:26.712070    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:01:36.711128 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:01:36.711078    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 16 20:01:46.721340 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:01:46.721305    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 20:01:46.729109 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:01:46.729087    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 20:02:13.828095 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:13.828062    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"]
Sep 16 20:02:28.850760 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:28.850700    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main" containerID="cri-o://f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7" gracePeriod=45
Sep 16 20:02:29.599550 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.599523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 20:02:29.740303 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.740264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a567a571-b2e9-41f0-82bc-d8d287474c49-tls-certs\") pod \"a567a571-b2e9-41f0-82bc-d8d287474c49\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") "
Sep 16 20:02:29.740303 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.740309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-tmp-dir\") pod \"a567a571-b2e9-41f0-82bc-d8d287474c49\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") "
Sep 16 20:02:29.740547 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.740334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4bgx\" (UniqueName: \"kubernetes.io/projected/a567a571-b2e9-41f0-82bc-d8d287474c49-kube-api-access-f4bgx\") pod \"a567a571-b2e9-41f0-82bc-d8d287474c49\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") "
Sep 16 20:02:29.740547 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.740360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a571-b2e9-41f0-82bc-d8d287474c49-cabundle-cert\") pod \"a567a571-b2e9-41f0-82bc-d8d287474c49\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") "
Sep 16 20:02:29.740547 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.740383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-dshm\") pod \"a567a571-b2e9-41f0-82bc-d8d287474c49\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") "
Sep 16 20:02:29.740547 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.740427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-home\") pod \"a567a571-b2e9-41f0-82bc-d8d287474c49\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") "
Sep 16 20:02:29.740547 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.740490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-model-cache\") pod \"a567a571-b2e9-41f0-82bc-d8d287474c49\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") "
Sep 16 20:02:29.740547 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.740530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-kserve-provision-location\") pod \"a567a571-b2e9-41f0-82bc-d8d287474c49\" (UID: \"a567a571-b2e9-41f0-82bc-d8d287474c49\") "
Sep 16 20:02:29.740979 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.740769    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a567a571-b2e9-41f0-82bc-d8d287474c49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a567a571-b2e9-41f0-82bc-d8d287474c49" (UID: "a567a571-b2e9-41f0-82bc-d8d287474c49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:02:29.741143 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.741114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-model-cache" (OuterVolumeSpecName: "model-cache") pod "a567a571-b2e9-41f0-82bc-d8d287474c49" (UID: "a567a571-b2e9-41f0-82bc-d8d287474c49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:29.741237 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.741130    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-home" (OuterVolumeSpecName: "home") pod "a567a571-b2e9-41f0-82bc-d8d287474c49" (UID: "a567a571-b2e9-41f0-82bc-d8d287474c49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:29.742857 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.742836    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-dshm" (OuterVolumeSpecName: "dshm") pod "a567a571-b2e9-41f0-82bc-d8d287474c49" (UID: "a567a571-b2e9-41f0-82bc-d8d287474c49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:29.743249 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.743223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a567a571-b2e9-41f0-82bc-d8d287474c49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a567a571-b2e9-41f0-82bc-d8d287474c49" (UID: "a567a571-b2e9-41f0-82bc-d8d287474c49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:02:29.743339 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.743292    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a567a571-b2e9-41f0-82bc-d8d287474c49-kube-api-access-f4bgx" (OuterVolumeSpecName: "kube-api-access-f4bgx") pod "a567a571-b2e9-41f0-82bc-d8d287474c49" (UID: "a567a571-b2e9-41f0-82bc-d8d287474c49"). InnerVolumeSpecName "kube-api-access-f4bgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:02:29.759038 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.758998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a567a571-b2e9-41f0-82bc-d8d287474c49" (UID: "a567a571-b2e9-41f0-82bc-d8d287474c49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:29.771286 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.771252    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a567a571-b2e9-41f0-82bc-d8d287474c49" (UID: "a567a571-b2e9-41f0-82bc-d8d287474c49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:29.841783 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.841747    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4bgx\" (UniqueName: \"kubernetes.io/projected/a567a571-b2e9-41f0-82bc-d8d287474c49-kube-api-access-f4bgx\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:02:29.841783 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.841779    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a571-b2e9-41f0-82bc-d8d287474c49-cabundle-cert\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:02:29.841783 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.841791    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:02:29.842023 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.841799    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:02:29.842023 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.841806    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:02:29.842023 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.841814    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-kserve-provision-location\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:02:29.842023 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.841824    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a567a571-b2e9-41f0-82bc-d8d287474c49-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:02:29.842023 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:29.841833    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a567a571-b2e9-41f0-82bc-d8d287474c49-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:02:30.411940 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.411883    2580 generic.go:356] "Generic (PLEG): container finished" podID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerID="f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7" exitCode=0
Sep 16 20:02:30.411940 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.411932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" event={"ID":"a567a571-b2e9-41f0-82bc-d8d287474c49","Type":"ContainerDied","Data":"f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7"}
Sep 16 20:02:30.412376 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.411952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"
Sep 16 20:02:30.412376 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.411993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj" event={"ID":"a567a571-b2e9-41f0-82bc-d8d287474c49","Type":"ContainerDied","Data":"6aa365e087ec53894faa40dd083a3ba93f5b4dc2f1e9f21866b4ad65647a0874"}
Sep 16 20:02:30.412376 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.412022    2580 scope.go:117] "RemoveContainer" containerID="f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7"
Sep 16 20:02:30.420715 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.420697    2580 scope.go:117] "RemoveContainer" containerID="a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3"
Sep 16 20:02:30.430974 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.430864    2580 scope.go:117] "RemoveContainer" containerID="f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7"
Sep 16 20:02:30.431538 ip-10-0-139-95 kubenswrapper[2580]: E0916 20:02:30.431505    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7\": container with ID starting with f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7 not found: ID does not exist" containerID="f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7"
Sep 16 20:02:30.431627 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.431545    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7"} err="failed to get container status \"f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7\": rpc error: code = NotFound desc = could not find container \"f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7\": container with ID starting with f003c038b45a67b674750119263fc90c0b9678b7e58b726c93b417783c764ec7 not found: ID does not exist"
Sep 16 20:02:30.431627 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.431569    2580 scope.go:117] "RemoveContainer" containerID="a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3"
Sep 16 20:02:30.431927 ip-10-0-139-95 kubenswrapper[2580]: E0916 20:02:30.431878    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3\": container with ID starting with a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3 not found: ID does not exist" containerID="a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3"
Sep 16 20:02:30.431994 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.431936    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3"} err="failed to get container status \"a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3\": rpc error: code = NotFound desc = could not find container \"a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3\": container with ID starting with a3c52d4365c43dfc872e7643b72c02052665b315a4fa3429dbc6137223bf23b3 not found: ID does not exist"
Sep 16 20:02:30.433056 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.433038    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"]
Sep 16 20:02:30.438334 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:30.438313    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d4566946d85rj"]
Sep 16 20:02:32.276180 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:32.276137    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" path="/var/lib/kubelet/pods/a567a571-b2e9-41f0-82bc-d8d287474c49/volumes"
Sep 16 20:02:36.916076 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.916045    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"]
Sep 16 20:02:36.916450 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.916385    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy"
Sep 16 20:02:36.916450 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.916396    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy"
Sep 16 20:02:36.916450 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.916405    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="storage-initializer"
Sep 16 20:02:36.916450 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.916410    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="storage-initializer"
Sep 16 20:02:36.916450 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.916425    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main"
Sep 16 20:02:36.916450 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.916431    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main"
Sep 16 20:02:36.916674 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.916497    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy"
Sep 16 20:02:36.916674 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.916510    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a567a571-b2e9-41f0-82bc-d8d287474c49" containerName="main"
Sep 16 20:02:36.918499 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.918482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:36.921846 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.921818    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-c294d\""
Sep 16 20:02:36.922014 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.921935    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:02:36.922014 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.921940    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:02:36.922014 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.921969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:02:36.933500 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:36.933475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"]
Sep 16 20:02:37.002397 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.002367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq4dl\" (UniqueName: \"kubernetes.io/projected/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-kube-api-access-wq4dl\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.002536 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.002430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.002536 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.002454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.002536 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.002476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.002536 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.002492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.002536 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.002511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.103346 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.103305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.103346 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.103348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.103553 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.103373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.103553 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.103483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.103553 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.103517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.103726 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.103591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq4dl\" (UniqueName: \"kubernetes.io/projected/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-kube-api-access-wq4dl\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.103785 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.103751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.103855 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.103795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.103971 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.103950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.105829 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.105811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.106097 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.106079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.117263 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.117238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq4dl\" (UniqueName: \"kubernetes.io/projected/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-kube-api-access-wq4dl\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-tcfs6\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.227973 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.227951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:37.500949 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:37.500923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"]
Sep 16 20:02:37.502298 ip-10-0-139-95 kubenswrapper[2580]: W0916 20:02:37.502275    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd627ca5_74b1_4e2f_9c62_f00e1a8aff8f.slice/crio-1dafe08144d9e7d9d290c5381d62f92ab6752b8565bad968db2b2f928013da56 WatchSource:0}: Error finding container 1dafe08144d9e7d9d290c5381d62f92ab6752b8565bad968db2b2f928013da56: Status 404 returned error can't find the container with id 1dafe08144d9e7d9d290c5381d62f92ab6752b8565bad968db2b2f928013da56
Sep 16 20:02:38.450130 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:38.450097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6" event={"ID":"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f","Type":"ContainerStarted","Data":"197f2e270134070c6c78c344fa599f472dabf93c95bcf5a8b52e65108632054e"}
Sep 16 20:02:38.450502 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:38.450143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6" event={"ID":"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f","Type":"ContainerStarted","Data":"1dafe08144d9e7d9d290c5381d62f92ab6752b8565bad968db2b2f928013da56"}
Sep 16 20:02:39.228552 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:39.228521    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:47.229096 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:47.229044    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:47.247969 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:47.242609    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:02:47.267990 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:47.267945    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6" podStartSLOduration=11.267931516 podStartE2EDuration="11.267931516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:02:38.47609534 +0000 UTC m=+4738.737772352" watchObservedRunningTime="2026-09-16 20:02:47.267931516 +0000 UTC m=+4747.529608528"
Sep 16 20:02:47.503908 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:02:47.503804    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:03:11.472958 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.472857    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"]
Sep 16 20:03:11.482561 ip-10-0-139-95 kubenswrapper[2580]: E0916 20:03:11.482523    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:03:11.482561 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:03:11.482561 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 20:03:11.482561 ip-10-0-139-95 kubenswrapper[2580]:  > containerID="197f2e270134070c6c78c344fa599f472dabf93c95bcf5a8b52e65108632054e" cmd=["/bin/sleep","15"]
Sep 16 20:03:11.482866 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.482581    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6" message="" err=<
Sep 16 20:03:11.482866 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:03:11.482866 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 20:03:11.482866 ip-10-0-139-95 kubenswrapper[2580]:  >
Sep 16 20:03:11.482866 ip-10-0-139-95 kubenswrapper[2580]: E0916 20:03:11.482598    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:03:11.482866 ip-10-0-139-95 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:03:11.482866 ip-10-0-139-95 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 16 20:03:11.482866 ip-10-0-139-95 kubenswrapper[2580]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6" podUID="cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" containerName="main" containerID="cri-o://197f2e270134070c6c78c344fa599f472dabf93c95bcf5a8b52e65108632054e"
Sep 16 20:03:11.482866 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.482642    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6" podUID="cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" containerName="main" containerID="cri-o://197f2e270134070c6c78c344fa599f472dabf93c95bcf5a8b52e65108632054e" gracePeriod=60
Sep 16 20:03:11.493023 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.492885    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6" podUID="cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 20:03:11.603319 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.603292    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" containerID="197f2e270134070c6c78c344fa599f472dabf93c95bcf5a8b52e65108632054e" exitCode=0
Sep 16 20:03:11.603449 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.603321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6" event={"ID":"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f","Type":"ContainerDied","Data":"197f2e270134070c6c78c344fa599f472dabf93c95bcf5a8b52e65108632054e"}
Sep 16 20:03:11.722849 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.722825    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:03:11.779466 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779404    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-home\") pod \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") "
Sep 16 20:03:11.779466 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779436    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq4dl\" (UniqueName: \"kubernetes.io/projected/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-kube-api-access-wq4dl\") pod \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") "
Sep 16 20:03:11.779466 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779464    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-model-cache\") pod \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") "
Sep 16 20:03:11.779682 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779540    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tmp-dir\") pod \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") "
Sep 16 20:03:11.779682 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779588    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tls-certs\") pod \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") "
Sep 16 20:03:11.779682 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779610    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-dshm\") pod \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\" (UID: \"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f\") "
Sep 16 20:03:11.779682 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-home" (OuterVolumeSpecName: "home") pod "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" (UID: "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:11.779840 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779771    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" (UID: "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:11.779840 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" (UID: "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:11.779974 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779954    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-model-cache\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:03:11.780019 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779983    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tmp-dir\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:03:11.780019 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.779997    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-home\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:03:11.781934 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.781882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" (UID: "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:11.782033 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.781965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-dshm" (OuterVolumeSpecName: "dshm") pod "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" (UID: "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:11.782033 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.781963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-kube-api-access-wq4dl" (OuterVolumeSpecName: "kube-api-access-wq4dl") pod "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" (UID: "cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f"). InnerVolumeSpecName "kube-api-access-wq4dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:11.881086 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.881054    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-tls-certs\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:03:11.881086 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.881083    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-dshm\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:03:11.881086 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:11.881094    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq4dl\" (UniqueName: \"kubernetes.io/projected/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f-kube-api-access-wq4dl\") on node \"ip-10-0-139-95.ec2.internal\" DevicePath \"\""
Sep 16 20:03:12.609122 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:12.609034    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"
Sep 16 20:03:12.609122 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:12.609040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6" event={"ID":"cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f","Type":"ContainerDied","Data":"1dafe08144d9e7d9d290c5381d62f92ab6752b8565bad968db2b2f928013da56"}
Sep 16 20:03:12.609631 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:12.609133    2580 scope.go:117] "RemoveContainer" containerID="197f2e270134070c6c78c344fa599f472dabf93c95bcf5a8b52e65108632054e"
Sep 16 20:03:12.627712 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:12.627687    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"]
Sep 16 20:03:12.635331 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:12.635310    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-tcfs6"]
Sep 16 20:03:14.275439 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:03:14.275408    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" path="/var/lib/kubelet/pods/cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f/volumes"
Sep 16 20:04:27.689231 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:27.689196    2580 ???:1] "http: TLS handshake error from 10.0.139.95:55054: EOF"
Sep 16 20:04:28.360461 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:28.360420    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:28.453326 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:28.453287    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:28.473195 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:28.473166    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:28.495070 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:28.495048    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:30.029985 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:30.029953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:30.329293 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:30.329192    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:30.358307 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:30.358282    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:30.375303 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:30.375286    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:32.138505 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:32.138419    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:32.260539 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:32.260512    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:32.299318 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:32.299291    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:32.325063 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:32.325041    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:33.831217 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:33.831179    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:33.961334 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:33.961293    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:33.978626 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:33.978604    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:33.993702 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:33.993683    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:35.733159 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:35.733125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:35.776351 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:35.776322    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:35.789212 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:35.789187    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:35.803322 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:35.803301    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:37.379646 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:37.379610    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:37.422734 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:37.422711    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:37.447328 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:37.447310    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:37.465150 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:37.465134    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:39.055650 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:39.055613    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:39.107542 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:39.107510    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:39.125506 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:39.125479    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:39.142072 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:39.142055    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:40.683657 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:40.683623    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:40.717795 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:40.717770    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:40.737219 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:40.737193    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:40.757908 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:40.757875    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:42.520463 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:42.520422    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:42.657633 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:42.657603    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:42.675532 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:42.675502    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:42.694723 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:42.694699    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:44.253198 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:44.253164    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:44.297164 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:44.297134    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:44.312492 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:44.312462    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:44.326181 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:44.326161    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:45.860870 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:45.860828    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:45.903159 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:45.903133    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:45.916761 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:45.916733    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:45.931187 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:45.931167    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:47.512743 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:47.512702    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:47.618200 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:47.618163    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:47.631119 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:47.631083    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:47.643775 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:47.643745    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:49.244621 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:49.244589    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:49.291172 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:49.291127    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:49.306489 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:49.306457    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:49.319665 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:49.319643    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:50.862687 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:50.862651    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:50.912599 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:50.912551    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:50.924969 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:50.924942    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:50.937973 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:50.937951    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:52.503885 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:52.503849    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:52.552447 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:52.552418    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:52.567125 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:52.567104    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:52.581960 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:52.581943    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:54.123868 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:54.123835    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:54.241141 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:54.241108    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:54.253810 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:54.253781    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:54.270739 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:54.270717    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:55.838990 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:55.838950    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:55.923519 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:55.923482    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:55.938362 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:55.938333    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:55.951811 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:55.951791    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:57.555706 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:57.555670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:57.603527 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:57.603500    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:57.621184 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:57.621164    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:57.638278 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:57.638262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:04:59.254086 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:59.254049    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:04:59.337378 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:59.337342    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:04:59.349960 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:59.349929    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:04:59.363881 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:04:59.363858    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:05:00.947668 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:00.947612    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:05:01.005169 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:01.005130    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:05:01.023207 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:01.023177    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:05:01.040254 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:01.040235    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:05:02.601630 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:02.601587    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:05:02.660057 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:02.660029    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:05:02.674343 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:02.674323    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:05:02.688704 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:02.688682    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:05:04.298397 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:04.298369    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:05:04.365210 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:04.365180    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:05:04.405092 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:04.405053    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:05:04.442111 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:04.442089    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:05:06.012815 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:06.012770    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:05:06.066064 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:06.066035    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:05:06.080439 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:06.080419    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:05:06.094119 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:06.094099    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:05:07.657092 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:07.657057    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:05:07.695010 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:07.694970    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:05:07.706978 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:07.706955    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:05:07.721685 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:07.721670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:05:09.274048 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:09.274015    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:05:09.315574 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:09.315549    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:05:09.330168 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:09.330145    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:05:09.344724 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:09.344706    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:05:14.735603 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:14.735576    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4cwkd_ed7a87c4-b91c-491e-9c5c-deab72585f4d/global-pull-secret-syncer/0.log"
Sep 16 20:05:14.941766 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:14.941731    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wwmwd_cb9bff8a-35f8-4491-a764-3e1318d4b95f/konnectivity-agent/0.log"
Sep 16 20:05:14.996647 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:14.996564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-95.ec2.internal_c92465bf8a81cf25e8e179fc079decff/haproxy/0.log"
Sep 16 20:05:19.889861 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:19.889828    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-rttvb_8de2c82b-97b2-4cf7-a5e7-8db239b6bdba/authorino/0.log"
Sep 16 20:05:19.940696 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:19.940668    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-kqh6t_9eea2e73-cb24-4d53-95f0-0b5fdc3b9648/manager/0.log"
Sep 16 20:05:19.984380 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:19.984347    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-svjrf_e64ece76-0bf8-4a9f-95eb-7869c05d70f7/manager/0.log"
Sep 16 20:05:20.012709 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:20.012692    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-92gtp_e3165ef0-0332-4646-acc9-05aa0b398780/kuadrant-console-plugin/0.log"
Sep 16 20:05:21.901016 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:21.900940    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cj7nd_aaa06985-4632-4f9f-bc05-e02d67eaa58c/kube-state-metrics/0.log"
Sep 16 20:05:21.939588 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:21.939558    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cj7nd_aaa06985-4632-4f9f-bc05-e02d67eaa58c/kube-rbac-proxy-main/0.log"
Sep 16 20:05:21.978733 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:21.978705    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cj7nd_aaa06985-4632-4f9f-bc05-e02d67eaa58c/kube-rbac-proxy-self/0.log"
Sep 16 20:05:22.012961 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:22.012937    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77f64d7c89-dsgzg_a31cc2ae-3760-4df3-9360-6fc14b749c30/metrics-server/0.log"
Sep 16 20:05:22.606649 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:22.606622    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t54ww_660f2c28-9fa2-4a58-8b34-b7eb6de0ab31/node-exporter/0.log"
Sep 16 20:05:22.663949 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:22.663845    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t54ww_660f2c28-9fa2-4a58-8b34-b7eb6de0ab31/kube-rbac-proxy/0.log"
Sep 16 20:05:22.693436 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:22.693411    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t54ww_660f2c28-9fa2-4a58-8b34-b7eb6de0ab31/init-textfile/0.log"
Sep 16 20:05:22.752212 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:22.752187    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-c74hj_c21038a3-a68a-4ab9-b004-ba1aa8c61b1f/kube-rbac-proxy-main/0.log"
Sep 16 20:05:22.810095 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:22.810063    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-c74hj_c21038a3-a68a-4ab9-b004-ba1aa8c61b1f/kube-rbac-proxy-self/0.log"
Sep 16 20:05:22.903960 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:22.903933    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-c74hj_c21038a3-a68a-4ab9-b004-ba1aa8c61b1f/openshift-state-metrics/0.log"
Sep 16 20:05:22.969223 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:22.969191    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc8b8cd7-65e6-4049-830b-6ee8b767c637/prometheus/0.log"
Sep 16 20:05:23.003691 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.003671    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc8b8cd7-65e6-4049-830b-6ee8b767c637/config-reloader/0.log"
Sep 16 20:05:23.032913 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.032864    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc8b8cd7-65e6-4049-830b-6ee8b767c637/thanos-sidecar/0.log"
Sep 16 20:05:23.061008 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.060982    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc8b8cd7-65e6-4049-830b-6ee8b767c637/kube-rbac-proxy-web/0.log"
Sep 16 20:05:23.090366 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.090344    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc8b8cd7-65e6-4049-830b-6ee8b767c637/kube-rbac-proxy/0.log"
Sep 16 20:05:23.116342 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.116321    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc8b8cd7-65e6-4049-830b-6ee8b767c637/kube-rbac-proxy-thanos/0.log"
Sep 16 20:05:23.144644 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.144621    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc8b8cd7-65e6-4049-830b-6ee8b767c637/init-config-reloader/0.log"
Sep 16 20:05:23.186336 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.186260    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-lct5x_3b71e4c7-9555-4ca6-bc67-fe363d681acd/prometheus-operator/0.log"
Sep 16 20:05:23.207838 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.207814    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-lct5x_3b71e4c7-9555-4ca6-bc67-fe363d681acd/kube-rbac-proxy/0.log"
Sep 16 20:05:23.278138 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.278113    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f4874c55c-xsf6t_eccd6d44-5310-4e2b-b843-7fb14835e261/telemeter-client/0.log"
Sep 16 20:05:23.311380 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.311360    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f4874c55c-xsf6t_eccd6d44-5310-4e2b-b843-7fb14835e261/reload/0.log"
Sep 16 20:05:23.338105 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.338071    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f4874c55c-xsf6t_eccd6d44-5310-4e2b-b843-7fb14835e261/kube-rbac-proxy/0.log"
Sep 16 20:05:23.373021 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.372999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d77cbb8bc-p92n6_38bc79e7-4ff8-472c-a51a-23d8c76f5281/thanos-query/0.log"
Sep 16 20:05:23.397838 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.397818    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d77cbb8bc-p92n6_38bc79e7-4ff8-472c-a51a-23d8c76f5281/kube-rbac-proxy-web/0.log"
Sep 16 20:05:23.432408 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.432389    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d77cbb8bc-p92n6_38bc79e7-4ff8-472c-a51a-23d8c76f5281/kube-rbac-proxy/0.log"
Sep 16 20:05:23.461089 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.461035    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d77cbb8bc-p92n6_38bc79e7-4ff8-472c-a51a-23d8c76f5281/prom-label-proxy/0.log"
Sep 16 20:05:23.487860 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.487823    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d77cbb8bc-p92n6_38bc79e7-4ff8-472c-a51a-23d8c76f5281/kube-rbac-proxy-rules/0.log"
Sep 16 20:05:23.513204 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:23.513175    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d77cbb8bc-p92n6_38bc79e7-4ff8-472c-a51a-23d8c76f5281/kube-rbac-proxy-metrics/0.log"
Sep 16 20:05:24.556038 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.556005    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"]
Sep 16 20:05:24.556521 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.556501    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy"
Sep 16 20:05:24.556593 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.556524    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy"
Sep 16 20:05:24.556593 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.556546    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" containerName="main"
Sep 16 20:05:24.556593 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.556554    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" containerName="main"
Sep 16 20:05:24.556747 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.556647    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd627ca5-74b1-4e2f-9c62-f00e1a8aff8f" containerName="main"
Sep 16 20:05:24.556747 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.556663    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d535f54-ea0a-4a70-b066-925933b03a4e" containerName="istio-proxy"
Sep 16 20:05:24.560027 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.560005    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.563090 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.563070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qnxnw\"/\"openshift-service-ca.crt\""
Sep 16 20:05:24.563191 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.563088    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qnxnw\"/\"kube-root-ca.crt\""
Sep 16 20:05:24.563808 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.563792    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qnxnw\"/\"default-dockercfg-kqdgr\""
Sep 16 20:05:24.569606 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.569586    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"]
Sep 16 20:05:24.634502 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.634477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4km4\" (UniqueName: \"kubernetes.io/projected/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-kube-api-access-f4km4\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.634630 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.634505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-proc\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.634630 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.634535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-sys\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.634630 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.634579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-podres\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.634630 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.634614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-lib-modules\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.735951 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.735925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4km4\" (UniqueName: \"kubernetes.io/projected/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-kube-api-access-f4km4\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.736064 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.735959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-proc\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.736064 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.736002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-sys\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.736064 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.736033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-podres\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.736064 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.736055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-proc\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.736256 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.736056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-lib-modules\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.736256 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.736121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-sys\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.736256 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.736178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-lib-modules\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.736256 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.736185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-podres\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.744321 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.744299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4km4\" (UniqueName: \"kubernetes.io/projected/d3f277f5-c32e-40f2-9e16-50edcec8b6f6-kube-api-access-f4km4\") pod \"perf-node-gather-daemonset-snqrn\" (UID: \"d3f277f5-c32e-40f2-9e16-50edcec8b6f6\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:24.869570 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:24.869520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:25.145731 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:25.145710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"]
Sep 16 20:05:25.148141 ip-10-0-139-95 kubenswrapper[2580]: W0916 20:05:25.148111    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd3f277f5_c32e_40f2_9e16_50edcec8b6f6.slice/crio-8f0a9cc78a69f26b84792a86f5d456152bf015576efef3dec3722b5c43893462 WatchSource:0}: Error finding container 8f0a9cc78a69f26b84792a86f5d456152bf015576efef3dec3722b5c43893462: Status 404 returned error can't find the container with id 8f0a9cc78a69f26b84792a86f5d456152bf015576efef3dec3722b5c43893462
Sep 16 20:05:25.234155 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:25.234127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn" event={"ID":"d3f277f5-c32e-40f2-9e16-50edcec8b6f6","Type":"ContainerStarted","Data":"6507d48035816ecf6f755790c6d10ac4f7fd31e4a6f16308397e9c9719ce472b"}
Sep 16 20:05:25.234259 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:25.234177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn" event={"ID":"d3f277f5-c32e-40f2-9e16-50edcec8b6f6","Type":"ContainerStarted","Data":"8f0a9cc78a69f26b84792a86f5d456152bf015576efef3dec3722b5c43893462"}
Sep 16 20:05:25.234358 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:25.234344    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:25.257865 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:25.257814    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn" podStartSLOduration=1.2578010929999999 podStartE2EDuration="1.257801093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:05:25.255424775 +0000 UTC m=+4905.517101785" watchObservedRunningTime="2026-09-16 20:05:25.257801093 +0000 UTC m=+4905.519478103"
Sep 16 20:05:27.574137 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:27.574106    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m7nbk_197179e5-e12b-4199-afd0-2789401dfc65/dns/0.log"
Sep 16 20:05:27.598850 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:27.598829    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m7nbk_197179e5-e12b-4199-afd0-2789401dfc65/kube-rbac-proxy/0.log"
Sep 16 20:05:27.735039 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:27.735014    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x9lhv_3e740f22-4534-4351-8405-41938e2ad52c/dns-node-resolver/0.log"
Sep 16 20:05:28.394257 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:28.394233    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4t8hn_c198208c-5766-492e-8ae7-40c2ea1a637c/node-ca/0.log"
Sep 16 20:05:30.458343 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:30.458311    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2srkz_c2c4e8a1-73bd-4c00-9cd8-0e4905e590c1/serve-healthcheck-canary/0.log"
Sep 16 20:05:31.114501 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:31.114466    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m6wrk_304d64f3-67ea-4996-bd67-1fc8f9c75a14/kube-rbac-proxy/0.log"
Sep 16 20:05:31.142035 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:31.141999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m6wrk_304d64f3-67ea-4996-bd67-1fc8f9c75a14/exporter/0.log"
Sep 16 20:05:31.168789 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:31.168764    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m6wrk_304d64f3-67ea-4996-bd67-1fc8f9c75a14/extractor/0.log"
Sep 16 20:05:31.248134 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:31.248113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-snqrn"
Sep 16 20:05:34.684566 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:34.684537    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-77db69dd7c-f4tg4_65d406dc-ed70-4c70-aa5e-305c5d9d8c16/manager/0.log"
Sep 16 20:05:34.713883 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:34.713855    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mps75_e5f79e12-d03f-4f2a-8e20-51124afe58c7/openshift-lws-operator/0.log"
Sep 16 20:05:36.099473 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:36.099446    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-vrtsf_d3c3ed55-c98f-45d5-bba6-3da076466364/server/0.log"
Sep 16 20:05:37.648603 ip-10-0-139-95 kubenswrapper[2580]: I0916 20:05:37.648572    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-nm8jm_74733630-8c0d-49f5-9b0f-3eaa9bbcdfa8/seaweedfs/0.log"