Sep 09 20:50:57.941065 ip-10-0-140-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:50:57.941199 ip-10-0-140-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:50:57.941282 ip-10-0-140-132 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:50:57.941798 ip-10-0-140-132 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:51:08.189959 ip-10-0-140-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:51:08.189974 ip-10-0-140-132 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4cf6b8450f5f4167b7e55ed197faed66 --
Sep 09 20:52:41.246389 ip-10-0-140-132 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:52:41.688703 ip-10-0-140-132 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:41.688703 ip-10-0-140-132 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:52:41.688703 ip-10-0-140-132 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:41.688703 ip-10-0-140-132 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:52:41.688703 ip-10-0-140-132 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:41.691177 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.691065    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:52:41.695397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695381    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:41.695397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695399    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695402    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695406    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695409    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695411    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695414    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695417    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695425    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695428    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695431    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695433    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695436    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695438    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695441    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695444    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695448    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695452    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695455    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695458    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:41.695463 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695460    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695464    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695467    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695469    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695472    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695475    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695477    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695480    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695482    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695485    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695487    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695490    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695492    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695495    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695497    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695500    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695502    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695504    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695507    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695509    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:41.695914 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695511    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695515    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695517    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695520    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695522    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695526    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695528    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695531    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695533    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695535    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695538    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695540    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695543    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695545    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695548    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695551    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695554    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695556    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695562    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:41.696409 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695567    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695571    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695575    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695578    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695580    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695583    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695585    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695588    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695590    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695593    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695596    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695598    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695600    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695603    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695606    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695609    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695611    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695615    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695617    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695620    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:41.696869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695622    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695625    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695627    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695629    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695632    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695634    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695637    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695641    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695643    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695646    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695649    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695651    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695654    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695656    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695659    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695661    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695664    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695666    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695669    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.695671    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:41.697397 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696383    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696390    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696393    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696397    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696400    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696403    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696406    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696410    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696413    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696416    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696420    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696422    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696425    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696428    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696431    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696433    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696435    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696438    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696441    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696445    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:41.697869 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696447    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696453    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696456    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696460    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696462    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696465    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696468    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696470    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696473    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696475    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696478    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696481    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696483    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696485    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696488    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696490    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696492    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696495    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696497    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696500    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:41.698498 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696502    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696505    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696508    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696510    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696513    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696515    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696517    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696520    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696522    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696525    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696527    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696532    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696536    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696540    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696545    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696548    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696550    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696553    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696555    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696558    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:41.699317 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696560    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696562    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696565    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696568    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696570    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696572    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696575    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696577    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696580    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696582    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696585    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696587    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696590    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696592    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696595    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696599    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696602    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696605    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696608    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:41.699876 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696611    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696613    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696616    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696619    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696624    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696626    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696629    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696631    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696634    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696637    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696640    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696642    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696645    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696647    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696650    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696652    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696655    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696657    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696659    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:41.700469 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.696662    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697057    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697062    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697065    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697068    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697071    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697074    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697076    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697080    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697082    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697085    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697088    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697090    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697093    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697096    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697098    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697116    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697121    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697124    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697126    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:41.700935 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697129    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697132    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697134    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697137    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697139    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697142    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697145    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697147    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697150    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697153    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697155    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697158    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697160    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697163    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697166    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697168    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697171    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697173    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697175    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697178    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:41.701493 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697181    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697183    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697186    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697189    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697191    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697194    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697196    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697199    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697202    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697207    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697209    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697212    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697215    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697217    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697219    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697222    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697225    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697227    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697230    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:41.702091 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697232    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697235    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697237    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697239    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697242    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697245    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697247    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697250    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697252    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697254    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697257    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697260    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697264    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697267    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697269    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697272    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697275    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697277    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697280    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697282    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:41.702691 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697285    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697290    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697295    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697299    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697301    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697305    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697308    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697310    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697313    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697316    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697319    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697322    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697324    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697327    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697329    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697332    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697334    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697337    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697339    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:41.703384 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697341    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.697344    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697408    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697439    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697445    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697449    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697453    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697457    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697461    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697466    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697469    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697472    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697475    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697478    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697481    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697486    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697489    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697492    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697495    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697498    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697502    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697504    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697507    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697510    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:52:41.703847 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697513    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697522    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697524    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697527    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697531    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697534    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697536    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697539    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697542    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697545    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697549    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697551    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697554    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697557    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697560    2583 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697562    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697567    2583 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697570    2583 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697573    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697578    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697581    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697584    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697587    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697591    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697594    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:52:41.704467 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697597    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697600    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697603    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697606    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697608    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697611    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697614    2583 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697617    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697620    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697623    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697626    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697629    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697632    2583 flags.go:64] FLAG: --help="false"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697634    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-140-132.ec2.internal"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697637    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697640    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697643    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697646    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697649    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697652    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697655    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697658    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697660    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:52:41.705059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697663    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697666    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697669    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697673    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697676    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697679    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697681    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697685    2583 flags.go:64] FLAG: --lock-file=""
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697688    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697691    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697694    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697699    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697702    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697705    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697707    2583 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697710    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697713    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697716    2583 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697718    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697723    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697725    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697730    2583 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697732    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697735    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697738    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:52:41.705738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697741    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697744    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697747    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697749    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697757    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697760    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697763    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697766    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697768    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697774    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697778    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697782    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697784    2583 flags.go:64] FLAG: --port="10250"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697787    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697790    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a1b39d12a05c87a3"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697795    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697798    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697801    2583 flags.go:64] FLAG: --register-node="true"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697803    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697807    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697810    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697812    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697815    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697819    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697821    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:52:41.706405 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697824    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697827    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697830    2583 flags.go:64] FLAG: --runonce="false"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697832    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697835    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697838    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697841    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697844    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697847    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697849    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697853    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697856    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697859    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697861    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697864    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697867    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697870    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697874    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697880    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697883    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697886    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697890    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697893    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697896    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697899    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:52:41.707134 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697901    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697904    2583 flags.go:64] FLAG: --v="2"
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697908    2583 flags.go:64] FLAG: --version="false"
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697911    2583 flags.go:64] FLAG: --vmodule=""
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697915    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.697918    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698733    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698752    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698757    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698762    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698767    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698771    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698778    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698785    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698796    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698800    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698805    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698809    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698813    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698817    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:41.707766 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698821    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698825    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698829    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698833    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698837    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698841    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698848    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698859    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698864    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698868    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698873    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698878    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698882    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698886    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698890    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698894    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698898    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698904    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698911    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:41.708285 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698924    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698929    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698934    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698938    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698942    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698946    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698950    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698954    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698958    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698963    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698967    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698971    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698976    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698984    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698989    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698993    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.698998    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699002    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699006    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699010    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:41.708751 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699014    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699017    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699022    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699026    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699030    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699039    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699043    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699047    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699051    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699055    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699059    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699063    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699067    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699072    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699076    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699080    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699084    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699089    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699098    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699129    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:41.709352 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699175    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699188    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699195    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699201    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699206    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699211    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699218    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699222    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699227    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699232    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699236    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699248    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699358    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699367    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699373    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699377    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699381    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699386    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699389    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699395    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:41.709904 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699400    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:41.710784 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699404    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:41.710784 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699408    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:41.710784 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699413    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:41.710784 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699418    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:41.710784 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.699422    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:41.710784 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.700178    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:41.711827 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.711805    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:52:41.711827 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.711826    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:52:41.712797 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.712786    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:52:41.712850 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.712801    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:52:41.712905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712897    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712906    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712911    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712916    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712920    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712924    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712930    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712934    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712938    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712942    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712946    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712950    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712957    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:41.712958 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712962    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712966    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712971    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712975    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712979    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712983    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712987    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712992    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.712996    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713000    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713004    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713008    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713011    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713015    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713019    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713023    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713027    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713031    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713035    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713039    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:41.713533 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713043    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713047    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713051    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713056    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713060    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713063    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713068    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713071    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713076    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713081    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713084    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713088    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713093    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713097    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713126    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713132    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713136    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713140    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713144    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713149    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:41.714437 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713153    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713159    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713165    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713170    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713174    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713179    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713183    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713188    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713192    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713196    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713200    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713204    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713208    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713212    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713216    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713221    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713224    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713229    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713233    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:41.714928 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713237    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713241    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713246    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713250    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713254    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713260    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713267    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713273    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713277    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713282    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713287    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713291    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713296    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713301    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713305    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713310    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713314    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713318    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713323    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:41.715680 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713327    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713331    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713335    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713339    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713344    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713348    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713352    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713356    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.713364    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713550    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713558    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713562    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713566    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713571    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713575    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:41.716465 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713579    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713583    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713588    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713596    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713603    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713608    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713613    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713617    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713622    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713626    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713630    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713634    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713638    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713642    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713646    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713650    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713654    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713657    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713661    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:41.716905 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713665    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713672    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713677    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713682    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713687    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713692    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713696    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713701    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713705    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713710    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713715    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713719    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713724    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713728    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713732    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713736    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713741    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713748    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713753    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:41.717460 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713758    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713762    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713767    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713771    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713775    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713778    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713782    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713786    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713790    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713795    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713799    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713803    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713807    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713811    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713815    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713819    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713823    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713828    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713832    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713836    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:41.718007 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713840    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713844    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713849    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713854    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713858    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713861    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713865    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713870    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713874    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713879    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713883    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713887    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713892    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713896    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713900    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713903    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713907    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713911    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713916    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713919    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713924    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:41.718589 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713928    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713932    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713936    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713940    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713944    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713947    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713951    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713955    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713959    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713963    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713968    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713972    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713975    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:41.713980    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:41.719178 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.713988    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:41.719540 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.714179    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:52:41.719540 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.717412    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:52:41.719540 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.718593    2583 server.go:1019] "Starting client certificate rotation"
Sep 09 20:52:41.719540 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.718703    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:41.719696 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.719605    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:41.722334 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.722316    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:41.723570 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.723547    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:41.735150 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.735129    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:52:41.738757 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.738737    2583 log.go:25] "Validated CRI v1 image API"
Sep 09 20:52:41.740275 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.740241    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:52:41.748038 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.748012    2583 fs.go:135] Filesystem UUIDs: map[6fd95e23-4195-4b21-b4bb-0866b3f3f414:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7e2469f9-8a3d-4015-9473-f487dce74241:/dev/nvme0n1p4]
Sep 09 20:52:41.748157 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.748035    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:52:41.753434 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.753299    2583 manager.go:216] Machine: {Timestamp:2026-09-09 20:52:41.751964424 +0000 UTC m=+0.376438356 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091192 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f97edb49d287fb7e7fe5f7eff754 SystemUUID:ec24f97e-db49-d287-fb7e-7fe5f7eff754 BootID:4cf6b845-0f5f-4167-b7e5-5ed197faed66 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ef:38:49:08:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ef:38:49:08:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:2d:f4:c2:ff:50 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:52:41.753434 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.753417    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:52:41.753672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.753545    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:52:41.754849 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.754815    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:52:41.754994 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.754852    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:52:41.755753 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.755742    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:52:41.755800 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.755755    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:52:41.755800 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.755763    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:41.756620 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.756610    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:41.757909 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.757899    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:52:41.762756 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.762743    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:41.762881 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.762871    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:52:41.762913 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.762908    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:52:41.762946 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.762923    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:52:41.762946 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.762939    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:52:41.762997 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.762948    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:52:41.762997 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.762957    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:52:41.763365 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.763352    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:41.763425 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.763371    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:41.766604 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.766576    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:41.767614 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.767598    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:52:41.770594 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.770579    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:52:41.770659 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.770610    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:52:41.772022 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772009    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:52:41.772022 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772025    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772031    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772036    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772044    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772049    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772062    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772067    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772073    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772079    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772088    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:52:41.772123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772094    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:52:41.773007 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.772997    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:52:41.773007 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.773007    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:52:41.774637 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.774610    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:52:41.774929 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.774900    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:52:41.776902 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.776883    2583 server.go:1268] "Started kubelet"
Sep 09 20:52:41.776998 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.776937    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:52:41.777054 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.776981    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:52:41.777090 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.777053    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:52:41.777704 ip-10-0-140-132 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:52:41.778672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.778586    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:52:41.779071 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.779056    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:52:41.782648 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.782616    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:52:41.784320 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.784302    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:52:41.785338 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.785318    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wl5gv"
Sep 09 20:52:41.785409 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.785364    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:41.786194 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.786173    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:52:41.786282 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.784335    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-132.ec2.internal.18d3c2585d718e73  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-132.ec2.internal,UID:ip-10-0-140-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-132.ec2.internal,},FirstTimestamp:2026-09-09 20:52:41.776852595 +0000 UTC m=+0.401326589,LastTimestamp:2026-09-09 20:52:41.776852595 +0000 UTC m=+0.401326589,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-132.ec2.internal,}"
Sep 09 20:52:41.787144 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.787094    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:52:41.787144 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.787145    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:52:41.787297 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.787195    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:52:41.788099 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.788082    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:41.788254 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.788240    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:52:41.788254 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.788254    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:52:41.788578 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.788560    2583 factory.go:153] Registering CRI-O factory
Sep 09 20:52:41.788657 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.788609    2583 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:52:41.788709 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.788662    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:52:41.788709 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.788672    2583 factory.go:55] Registering systemd factory
Sep 09 20:52:41.788709 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.788681    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:52:41.788709 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.788698    2583 factory.go:103] Registering Raw factory
Sep 09 20:52:41.788871 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.788723    2583 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:52:41.790606 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.790415    2583 manager.go:349] Starting recovery of all containers
Sep 09 20:52:41.791006 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.790975    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:52:41.791227 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.791204    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:52:41.794002 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.793979    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wl5gv"
Sep 09 20:52:41.800722 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.800704    2583 manager.go:354] Recovery completed
Sep 09 20:52:41.805059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.805046    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:41.807816 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.807796    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:41.807889 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.807852    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:41.807889 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.807864    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:41.808617 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.808605    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:52:41.808617 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.808616    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:52:41.808721 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.808626    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:41.808721 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.808634    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:41.809538 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.809522    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:41.809590 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.809549    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:41.809590 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.809558    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:41.810377 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.810293    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-132.ec2.internal.18d3c2585f4a0632  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-132.ec2.internal,UID:ip-10-0-140-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-132.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-132.ec2.internal,},FirstTimestamp:2026-09-09 20:52:41.807816242 +0000 UTC m=+0.432290174,LastTimestamp:2026-09-09 20:52:41.807816242 +0000 UTC m=+0.432290174,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-132.ec2.internal,}"
Sep 09 20:52:41.810785 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.810772    2583 policy_none.go:49] "None policy: Start"
Sep 09 20:52:41.810832 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.810788    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:52:41.810832 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.810798    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:52:41.812842 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.812831    2583 policy_none.go:47] "Start"
Sep 09 20:52:41.849783 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.849753    2583 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.849870    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.849884    2583 server.go:87] "Starting device plugin registration server"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.850180    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.850198    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.850275    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.850358    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.850369    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.851038    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.851130    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.857465    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.858840    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.858884    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.858904    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.858954    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:52:41.867989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.861080    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:41.950626 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.950534    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:41.951556 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.951537    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:41.951652 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.951572    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:41.951652 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.951582    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:41.951652 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.951604    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-132.ec2.internal"
Sep 09 20:52:41.959179 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.959157    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"]
Sep 09 20:52:41.959241 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.959232    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:41.961311 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.961285    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:41.961436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.961319    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:41.961436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.961334    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:41.962019 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.961996    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-132.ec2.internal"
Sep 09 20:52:41.962143 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.962049    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-132.ec2.internal\": node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:41.963814 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.963796    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:41.963936 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.963921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:41.963990 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.963949    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:41.964590 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.964574    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:41.964636 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.964607    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:41.964636 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.964609    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:41.964636 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.964622    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:41.964636 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.964627    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:41.964754 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.964640    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:41.966903 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.966885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:41.967005 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.966909    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:41.967647 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.967628    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:41.967713 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.967664    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:41.967713 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.967679    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:41.980543 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.980518    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:41.988815 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.988792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fdee4d29f0b1359d09bd37b4c77420b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" (UID: \"9fdee4d29f0b1359d09bd37b4c77420b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:41.988899 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.988818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83c08e0d52b4ee9262822078dda7d62a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"83c08e0d52b4ee9262822078dda7d62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:41.988899 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:41.988835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83c08e0d52b4ee9262822078dda7d62a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"83c08e0d52b4ee9262822078dda7d62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:41.991820 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.991803    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-132.ec2.internal\" not found" node="ip-10-0-140-132.ec2.internal"
Sep 09 20:52:41.996264 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:41.996248    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-132.ec2.internal\" not found" node="ip-10-0-140-132.ec2.internal"
Sep 09 20:52:42.081156 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.081122    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:42.088955 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.088927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83c08e0d52b4ee9262822078dda7d62a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"83c08e0d52b4ee9262822078dda7d62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:42.088955 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.088942    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83c08e0d52b4ee9262822078dda7d62a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"83c08e0d52b4ee9262822078dda7d62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:42.089143 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.088978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83c08e0d52b4ee9262822078dda7d62a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"83c08e0d52b4ee9262822078dda7d62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:42.089143 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.089001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fdee4d29f0b1359d09bd37b4c77420b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" (UID: \"9fdee4d29f0b1359d09bd37b4c77420b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:42.089143 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.089059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fdee4d29f0b1359d09bd37b4c77420b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" (UID: \"9fdee4d29f0b1359d09bd37b4c77420b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:42.089143 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.089087    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83c08e0d52b4ee9262822078dda7d62a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"83c08e0d52b4ee9262822078dda7d62a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:42.181889 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.181851    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:42.282761 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.282688    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:42.295243 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.295216    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:42.301795 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.300715    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:42.383437 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.383252    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:42.483864 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.483830    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:42.584461 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.584417    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:42.685079 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.685045    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:42.718616 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.718583    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:52:42.719077 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.718791    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:52:42.785097 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.785068    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:42.785247 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.785165    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:42.804180 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.804143    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:47:42 +0000 UTC" deadline="2028-05-26 04:03:12.98350429 +0000 UTC"
Sep 09 20:52:42.804180 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.804173    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14983h10m30.179333958s"
Sep 09 20:52:42.885748 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.885673    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:42.968562 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:42.968511    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c08e0d52b4ee9262822078dda7d62a.slice/crio-eee3c34e2559b4140fb656e0f3197f28bdc75819499d91862505b022b3593842 WatchSource:0}: Error finding container eee3c34e2559b4140fb656e0f3197f28bdc75819499d91862505b022b3593842: Status 404 returned error can't find the container with id eee3c34e2559b4140fb656e0f3197f28bdc75819499d91862505b022b3593842
Sep 09 20:52:42.968919 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:42.968896    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdee4d29f0b1359d09bd37b4c77420b.slice/crio-0870c0d6e101dcd826495fe21629fccf01510b8d0e52dee534932dbc97ffdbc6 WatchSource:0}: Error finding container 0870c0d6e101dcd826495fe21629fccf01510b8d0e52dee534932dbc97ffdbc6: Status 404 returned error can't find the container with id 0870c0d6e101dcd826495fe21629fccf01510b8d0e52dee534932dbc97ffdbc6
Sep 09 20:52:42.972805 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:42.972787    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:52:42.985989 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:42.985969    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:43.086876 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.086844    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Sep 09 20:52:43.135947 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.135858    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:43.136216 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.136199    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:43.186994 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.186953    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:43.199904 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.199880    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:43.201033 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.201019    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Sep 09 20:52:43.211617 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.211589    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:43.764484 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.764168    2583 apiserver.go:52] "Watching apiserver"
Sep 09 20:52:43.772113 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.772084    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:52:43.772739 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.772711    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-mmcfd","openshift-multus/multus-additional-cni-plugins-p7m7l","openshift-multus/multus-m92df","openshift-multus/network-metrics-daemon-29hf4","openshift-network-diagnostics/network-check-target-q2w92","kube-system/global-pull-secret-syncer-nrhbs","kube-system/konnectivity-agent-k6zp7","openshift-cluster-node-tuning-operator/tuned-gbxrx","openshift-image-registry/node-ca-mxtw2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal","openshift-network-operator/iptables-alerter-vp5cx","openshift-ovn-kubernetes/ovnkube-node-8nfkh","kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"]
Sep 09 20:52:43.775579 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.775554    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:52:43.778007 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.777986    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.778527 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.778507    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:52:43.780160 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.780091    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4zzvr\""
Sep 09 20:52:43.780444 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.780427    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:52:43.780818 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.780795    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-64qrd\""
Sep 09 20:52:43.781025 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.781010    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:43.781306 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.781291    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:43.782495 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.782474    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.782841 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.782819    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:52:43.784866 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.784813    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:43.785149 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.785092    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:52:43.785239 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.785166    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:52:43.785782 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.785759    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rpp6z\""
Sep 09 20:52:43.785902 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.785889    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:52:43.786075 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.786058    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:43.786218 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.786170    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:52:43.787313 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.787294    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:43.787414 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.787376    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:52:43.787487 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.787426    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:43.787551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.787471    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:52:43.790092 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.790075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:43.790214 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.790175    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:52:43.792673 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.792654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:43.795220 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.795199    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:43.795463 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.795443    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zt2cb\""
Sep 09 20:52:43.795683 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.795660    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:52:43.796611 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.795870    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:52:43.796611 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.796081    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:52:43.798221 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.797459    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:43.798221 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.797976    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:52:43.798400 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.798227    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qfkpq\""
Sep 09 20:52:43.798400 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.798371    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:52:43.798499 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.798449    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-hostroot\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02275490-d835-48ad-836f-4eb605bd4d0f-multus-daemon-config\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dad8f34d-a7d6-4549-ac85-b9f9e0589a5a-iptables-alerter-script\") pod \"iptables-alerter-vp5cx\" (UID: \"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a\") " pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-cnibin\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62c0bc3b-be18-47e4-bd35-d90287e22c4b-konnectivity-ca\") pod \"konnectivity-agent-k6zp7\" (UID: \"62c0bc3b-be18-47e4-bd35-d90287e22c4b\") " pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjkl\" (UniqueName: \"kubernetes.io/projected/d43fba51-6814-4c9e-a5c8-a1dd796b7386-kube-api-access-jjjkl\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2685e808-1a0b-4fb0-b19a-876ead4ad42e-kubelet-config\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62c0bc3b-be18-47e4-bd35-d90287e22c4b-agent-certs\") pod \"konnectivity-agent-k6zp7\" (UID: \"62c0bc3b-be18-47e4-bd35-d90287e22c4b\") " pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-system-cni-dir\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-multus-cni-dir\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800385    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-var-lib-cni-bin\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800391    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7wx6b\""
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dad8f34d-a7d6-4549-ac85-b9f9e0589a5a-host-slash\") pod \"iptables-alerter-vp5cx\" (UID: \"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a\") " pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-multus-socket-dir-parent\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-multus-conf-dir\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.801662 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02275490-d835-48ad-836f-4eb605bd4d0f-cni-binary-copy\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-var-lib-cni-multus\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-var-lib-kubelet\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-etc-kubernetes\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2685e808-1a0b-4fb0-b19a-876ead4ad42e-dbus\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkdq\" (UniqueName: \"kubernetes.io/projected/dad8f34d-a7d6-4549-ac85-b9f9e0589a5a-kube-api-access-zgkdq\") pod \"iptables-alerter-vp5cx\" (UID: \"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a\") " pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-run-netns\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800700    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-run-multus-certs\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlpj2\" (UniqueName: \"kubernetes.io/projected/02275490-d835-48ad-836f-4eb605bd4d0f-kube-api-access-dlpj2\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.800815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-run-k8s-cni-cncf-io\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.801291    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:52:43.802747 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.801409    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:52:43.803740 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.803710    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:43.807066 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.807045    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.809824 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.809802    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:43.809824 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.809819    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8bnxn\""
Sep 09 20:52:43.809978 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.809888    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:43.811128 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.811089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:43.813883 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.813683    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:52:43.813883 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.813775    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bpqbc\""
Sep 09 20:52:43.813883 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.813779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.815240 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.815218    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:52:43.816076 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.816054    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:52:43.816650 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.816626    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvs95\""
Sep 09 20:52:43.816942 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.816909    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:52:43.817352 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.817087    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:52:43.817352 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.817297    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:52:43.818154 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.817757    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:52:43.818154 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.817903    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:52:43.828901 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.828876    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rqx9n"
Sep 09 20:52:43.838590 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.838569    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rqx9n"
Sep 09 20:52:43.869606 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.869561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" event={"ID":"83c08e0d52b4ee9262822078dda7d62a","Type":"ContainerStarted","Data":"eee3c34e2559b4140fb656e0f3197f28bdc75819499d91862505b022b3593842"}
Sep 09 20:52:43.871151 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.871127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal" event={"ID":"9fdee4d29f0b1359d09bd37b4c77420b","Type":"ContainerStarted","Data":"0870c0d6e101dcd826495fe21629fccf01510b8d0e52dee534932dbc97ffdbc6"}
Sep 09 20:52:43.888135 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.888110    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:52:43.901061 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dad8f34d-a7d6-4549-ac85-b9f9e0589a5a-iptables-alerter-script\") pod \"iptables-alerter-vp5cx\" (UID: \"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a\") " pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.901257 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:43.901257 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-systemd\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.901257 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-slash\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.901257 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f8b5d92-d112-4bbc-a824-b4a96710d5db-ovnkube-config\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.901436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c58b4f94-071e-400c-bc9b-e72bb86a324f-cnibin\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:43.901436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-var-lib-kubelet\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.901436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f256da54-df5c-4dbc-9f2f-09b35844ea45-tmp\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.901436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901319    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-kubelet\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.901436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-run-systemd\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.901436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-system-cni-dir\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.901436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-multus-cni-dir\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901494    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-system-cni-dir\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-var-lib-cni-bin\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-socket-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c58b4f94-071e-400c-bc9b-e72bb86a324f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-var-lib-cni-bin\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dad8f34d-a7d6-4549-ac85-b9f9e0589a5a-host-slash\") pod \"iptables-alerter-vp5cx\" (UID: \"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a\") " pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-multus-cni-dir\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901630    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dad8f34d-a7d6-4549-ac85-b9f9e0589a5a-iptables-alerter-script\") pod \"iptables-alerter-vp5cx\" (UID: \"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a\") " pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c468bb92-a81e-4d78-88c0-ae85f7c69561-host\") pod \"node-ca-mxtw2\" (UID: \"c468bb92-a81e-4d78-88c0-ae85f7c69561\") " pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dad8f34d-a7d6-4549-ac85-b9f9e0589a5a-host-slash\") pod \"iptables-alerter-vp5cx\" (UID: \"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a\") " pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-var-lib-openvswitch\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-lib-modules\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.901762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-tuned\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02275490-d835-48ad-836f-4eb605bd4d0f-cni-binary-copy\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-var-lib-kubelet\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-device-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-sys-fs\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c468bb92-a81e-4d78-88c0-ae85f7c69561-serviceca\") pod \"node-ca-mxtw2\" (UID: \"c468bb92-a81e-4d78-88c0-ae85f7c69561\") " pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-var-lib-kubelet\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.901979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7p5t\" (UniqueName: \"kubernetes.io/projected/c468bb92-a81e-4d78-88c0-ae85f7c69561-kube-api-access-v7p5t\") pod \"node-ca-mxtw2\" (UID: \"c468bb92-a81e-4d78-88c0-ae85f7c69561\") " pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-run-netns\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-run-multus-certs\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-run-netns\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902107    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-run-multus-certs\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-sys\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902181    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-node-log\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-run-ovn-kubernetes\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.902447 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.902239    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.902336    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret podName:2685e808-1a0b-4fb0-b19a-876ead4ad42e nodeName:}" failed. No retries permitted until 2026-09-09 20:52:44.402296043 +0000 UTC m=+3.026769982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret") pod "global-pull-secret-syncer-nrhbs" (UID: "2685e808-1a0b-4fb0-b19a-876ead4ad42e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-cni-bin\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-run-k8s-cni-cncf-io\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-registration-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02275490-d835-48ad-836f-4eb605bd4d0f-cni-binary-copy\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c58b4f94-071e-400c-bc9b-e72bb86a324f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-run-k8s-cni-cncf-io\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902478    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-kubernetes\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902533    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4ln\" (UniqueName: \"kubernetes.io/projected/0f8b5d92-d112-4bbc-a824-b4a96710d5db-kube-api-access-sg4ln\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-hostroot\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c58b4f94-071e-400c-bc9b-e72bb86a324f-system-cni-dir\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-hostroot\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c58b4f94-071e-400c-bc9b-e72bb86a324f-cni-binary-copy\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvlj\" (UniqueName: \"kubernetes.io/projected/c58b4f94-071e-400c-bc9b-e72bb86a324f-kube-api-access-mjvlj\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:43.903394 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.902708    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-log-socket\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.902769    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs podName:d43fba51-6814-4c9e-a5c8-a1dd796b7386 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:44.4027531 +0000 UTC m=+3.027227034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs") pod "network-metrics-daemon-29hf4" (UID: "d43fba51-6814-4c9e-a5c8-a1dd796b7386") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-cnibin\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c58b4f94-071e-400c-bc9b-e72bb86a324f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62c0bc3b-be18-47e4-bd35-d90287e22c4b-konnectivity-ca\") pod \"konnectivity-agent-k6zp7\" (UID: \"62c0bc3b-be18-47e4-bd35-d90287e22c4b\") " pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-run\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902880    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-cnibin\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjkl\" (UniqueName: \"kubernetes.io/projected/d43fba51-6814-4c9e-a5c8-a1dd796b7386-kube-api-access-jjjkl\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.902930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2685e808-1a0b-4fb0-b19a-876ead4ad42e-kubelet-config\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62c0bc3b-be18-47e4-bd35-d90287e22c4b-agent-certs\") pod \"konnectivity-agent-k6zp7\" (UID: \"62c0bc3b-be18-47e4-bd35-d90287e22c4b\") " pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2685e808-1a0b-4fb0-b19a-876ead4ad42e-kubelet-config\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-host\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fvrf\" (UniqueName: \"kubernetes.io/projected/f256da54-df5c-4dbc-9f2f-09b35844ea45-kube-api-access-6fvrf\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5c2c926-9449-4539-b439-06fb5d910856-hosts-file\") pod \"node-resolver-mmcfd\" (UID: \"b5c2c926-9449-4539-b439-06fb5d910856\") " pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:43.904073 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-multus-socket-dir-parent\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-multus-conf-dir\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c2c926-9449-4539-b439-06fb5d910856-tmp-dir\") pod \"node-resolver-mmcfd\" (UID: \"b5c2c926-9449-4539-b439-06fb5d910856\") " pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903264    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-multus-socket-dir-parent\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqlgw\" (UniqueName: \"kubernetes.io/projected/b5c2c926-9449-4539-b439-06fb5d910856-kube-api-access-kqlgw\") pod \"node-resolver-mmcfd\" (UID: \"b5c2c926-9449-4539-b439-06fb5d910856\") " pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903310    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-multus-conf-dir\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-systemd-units\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f8b5d92-d112-4bbc-a824-b4a96710d5db-env-overrides\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903363    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-var-lib-cni-multus\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-etc-kubernetes\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903430    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62c0bc3b-be18-47e4-bd35-d90287e22c4b-konnectivity-ca\") pod \"konnectivity-agent-k6zp7\" (UID: \"62c0bc3b-be18-47e4-bd35-d90287e22c4b\") " pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2685e808-1a0b-4fb0-b19a-876ead4ad42e-dbus\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-sysconfig\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-host-var-lib-cni-multus\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903499    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-run-openvswitch\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02275490-d835-48ad-836f-4eb605bd4d0f-etc-kubernetes\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2685e808-1a0b-4fb0-b19a-876ead4ad42e-dbus\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:43.904672 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-cni-netd\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f8b5d92-d112-4bbc-a824-b4a96710d5db-ovnkube-script-lib\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkdq\" (UniqueName: \"kubernetes.io/projected/dad8f34d-a7d6-4549-ac85-b9f9e0589a5a-kube-api-access-zgkdq\") pod \"iptables-alerter-vp5cx\" (UID: \"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a\") " pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlpj2\" (UniqueName: \"kubernetes.io/projected/02275490-d835-48ad-836f-4eb605bd4d0f-kube-api-access-dlpj2\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m62q\" (UniqueName: \"kubernetes.io/projected/aba46778-09e3-41cb-a6dc-c26359982ef3-kube-api-access-6m62q\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f8b5d92-d112-4bbc-a824-b4a96710d5db-ovn-node-metrics-cert\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-modprobe-d\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-sysctl-d\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903776    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-sysctl-conf\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-etc-openvswitch\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-run-ovn\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02275490-d835-48ad-836f-4eb605bd4d0f-multus-daemon-config\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.903878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-run-netns\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:43.905551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.904317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02275490-d835-48ad-836f-4eb605bd4d0f-multus-daemon-config\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.908255 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.908230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62c0bc3b-be18-47e4-bd35-d90287e22c4b-agent-certs\") pod \"konnectivity-agent-k6zp7\" (UID: \"62c0bc3b-be18-47e4-bd35-d90287e22c4b\") " pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:52:43.909650 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.909628    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:43.909650 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.909652    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:43.909875 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.909661    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gflfq for pod openshift-network-diagnostics/network-check-target-q2w92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:43.909875 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:43.909699    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq podName:f982e35b-5816-44a8-92a2-8b363ebe4c36 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:44.409687345 +0000 UTC m=+3.034161266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gflfq" (UniqueName: "kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq") pod "network-check-target-q2w92" (UID: "f982e35b-5816-44a8-92a2-8b363ebe4c36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:43.913621 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.913598    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkdq\" (UniqueName: \"kubernetes.io/projected/dad8f34d-a7d6-4549-ac85-b9f9e0589a5a-kube-api-access-zgkdq\") pod \"iptables-alerter-vp5cx\" (UID: \"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a\") " pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:43.914292 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.914271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlpj2\" (UniqueName: \"kubernetes.io/projected/02275490-d835-48ad-836f-4eb605bd4d0f-kube-api-access-dlpj2\") pod \"multus-m92df\" (UID: \"02275490-d835-48ad-836f-4eb605bd4d0f\") " pod="openshift-multus/multus-m92df"
Sep 09 20:52:43.916836 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:43.916810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjkl\" (UniqueName: \"kubernetes.io/projected/d43fba51-6814-4c9e-a5c8-a1dd796b7386-kube-api-access-jjjkl\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:44.004208 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c58b4f94-071e-400c-bc9b-e72bb86a324f-system-cni-dir\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.004208 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c58b4f94-071e-400c-bc9b-e72bb86a324f-cni-binary-copy\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.004464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvlj\" (UniqueName: \"kubernetes.io/projected/c58b4f94-071e-400c-bc9b-e72bb86a324f-kube-api-access-mjvlj\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.004464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-log-socket\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.004464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c58b4f94-071e-400c-bc9b-e72bb86a324f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.004464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.004464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-run\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.004464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-host\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.004464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fvrf\" (UniqueName: \"kubernetes.io/projected/f256da54-df5c-4dbc-9f2f-09b35844ea45-kube-api-access-6fvrf\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.004464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5c2c926-9449-4539-b439-06fb5d910856-hosts-file\") pod \"node-resolver-mmcfd\" (UID: \"b5c2c926-9449-4539-b439-06fb5d910856\") " pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:44.004464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c2c926-9449-4539-b439-06fb5d910856-tmp-dir\") pod \"node-resolver-mmcfd\" (UID: \"b5c2c926-9449-4539-b439-06fb5d910856\") " pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:44.004927 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqlgw\" (UniqueName: \"kubernetes.io/projected/b5c2c926-9449-4539-b439-06fb5d910856-kube-api-access-kqlgw\") pod \"node-resolver-mmcfd\" (UID: \"b5c2c926-9449-4539-b439-06fb5d910856\") " pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:44.004927 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-systemd-units\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.004927 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f8b5d92-d112-4bbc-a824-b4a96710d5db-env-overrides\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.004927 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004560    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-systemd-units\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.004927 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-sysconfig\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.004927 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-log-socket\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.004927 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-sysconfig\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.004927 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-run-openvswitch\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.004927 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.004942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-cni-netd\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f8b5d92-d112-4bbc-a824-b4a96710d5db-ovnkube-script-lib\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m62q\" (UniqueName: \"kubernetes.io/projected/aba46778-09e3-41cb-a6dc-c26359982ef3-kube-api-access-6m62q\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f8b5d92-d112-4bbc-a824-b4a96710d5db-ovn-node-metrics-cert\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-modprobe-d\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-sysctl-d\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-sysctl-conf\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-etc-openvswitch\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.005415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-run-ovn\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.005931 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-run-netns\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.005931 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.005931 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005678    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.005931 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c58b4f94-071e-400c-bc9b-e72bb86a324f-cni-binary-copy\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.005931 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-run-ovn\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.005931 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-sysctl-d\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.006271 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.005981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-etc-openvswitch\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.006271 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006061    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-run-netns\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.006271 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f8b5d92-d112-4bbc-a824-b4a96710d5db-ovnkube-script-lib\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.006434 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-sysctl-conf\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.006680 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006658    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-modprobe-d\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.006746 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c58b4f94-071e-400c-bc9b-e72bb86a324f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.006746 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-run-openvswitch\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.006854 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006782    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-cni-netd\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.006854 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.006854 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-systemd\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.007006 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-slash\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.007006 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f8b5d92-d112-4bbc-a824-b4a96710d5db-ovnkube-config\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.007006 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c58b4f94-071e-400c-bc9b-e72bb86a324f-cnibin\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.007006 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-var-lib-kubelet\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.007006 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f256da54-df5c-4dbc-9f2f-09b35844ea45-tmp\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.007006 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-kubelet\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.007006 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c2c926-9449-4539-b439-06fb5d910856-tmp-dir\") pod \"node-resolver-mmcfd\" (UID: \"b5c2c926-9449-4539-b439-06fb5d910856\") " pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:44.007006 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.006996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-run-systemd\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-run-systemd\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-socket-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c58b4f94-071e-400c-bc9b-e72bb86a324f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c468bb92-a81e-4d78-88c0-ae85f7c69561-host\") pod \"node-ca-mxtw2\" (UID: \"c468bb92-a81e-4d78-88c0-ae85f7c69561\") " pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-var-lib-openvswitch\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-lib-modules\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-tuned\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007323    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c58b4f94-071e-400c-bc9b-e72bb86a324f-system-cni-dir\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-device-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-systemd\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-sys-fs\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.007432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c468bb92-a81e-4d78-88c0-ae85f7c69561-serviceca\") pod \"node-ca-mxtw2\" (UID: \"c468bb92-a81e-4d78-88c0-ae85f7c69561\") " pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:44.008051 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7p5t\" (UniqueName: \"kubernetes.io/projected/c468bb92-a81e-4d78-88c0-ae85f7c69561-kube-api-access-v7p5t\") pod \"node-ca-mxtw2\" (UID: \"c468bb92-a81e-4d78-88c0-ae85f7c69561\") " pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:44.008051 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-sys-fs\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.008051 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007745    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-var-lib-openvswitch\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.008051 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-lib-modules\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.008051 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-run\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.008051 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.007928    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-host\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.008385 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5c2c926-9449-4539-b439-06fb5d910856-hosts-file\") pod \"node-resolver-mmcfd\" (UID: \"b5c2c926-9449-4539-b439-06fb5d910856\") " pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:44.008385 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f8b5d92-d112-4bbc-a824-b4a96710d5db-env-overrides\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.008385 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-socket-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.008540 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c468bb92-a81e-4d78-88c0-ae85f7c69561-host\") pod \"node-ca-mxtw2\" (UID: \"c468bb92-a81e-4d78-88c0-ae85f7c69561\") " pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:44.008540 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-sys\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.008540 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-node-log\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.008637 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-run-ovn-kubernetes\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.008637 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-cni-bin\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.008707 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008656    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-device-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.008744 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008717    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-cni-bin\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.008776 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c58b4f94-071e-400c-bc9b-e72bb86a324f-cnibin\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.008850 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-sys\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.008894 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008881    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-node-log\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.008940 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.008927    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-run-ovn-kubernetes\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.009049 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-var-lib-kubelet\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.009085 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c58b4f94-071e-400c-bc9b-e72bb86a324f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.009119 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-slash\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.009119 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-registration-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.009206 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c58b4f94-071e-400c-bc9b-e72bb86a324f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.009206 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-kubernetes\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.009311 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4ln\" (UniqueName: \"kubernetes.io/projected/0f8b5d92-d112-4bbc-a824-b4a96710d5db-kube-api-access-sg4ln\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.009507 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009484    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8b5d92-d112-4bbc-a824-b4a96710d5db-host-kubelet\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.009624 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-kubernetes\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.009692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009649    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f8b5d92-d112-4bbc-a824-b4a96710d5db-ovnkube-config\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.009692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aba46778-09e3-41cb-a6dc-c26359982ef3-registration-dir\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.010240 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.009864    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c58b4f94-071e-400c-bc9b-e72bb86a324f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.011220 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.010759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c468bb92-a81e-4d78-88c0-ae85f7c69561-serviceca\") pod \"node-ca-mxtw2\" (UID: \"c468bb92-a81e-4d78-88c0-ae85f7c69561\") " pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:44.012417 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.010964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f256da54-df5c-4dbc-9f2f-09b35844ea45-tmp\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.012607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.012573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f256da54-df5c-4dbc-9f2f-09b35844ea45-etc-tuned\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.012675 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.012610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f8b5d92-d112-4bbc-a824-b4a96710d5db-ovn-node-metrics-cert\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.016956 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.016903    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqlgw\" (UniqueName: \"kubernetes.io/projected/b5c2c926-9449-4539-b439-06fb5d910856-kube-api-access-kqlgw\") pod \"node-resolver-mmcfd\" (UID: \"b5c2c926-9449-4539-b439-06fb5d910856\") " pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:44.017474 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.017422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7p5t\" (UniqueName: \"kubernetes.io/projected/c468bb92-a81e-4d78-88c0-ae85f7c69561-kube-api-access-v7p5t\") pod \"node-ca-mxtw2\" (UID: \"c468bb92-a81e-4d78-88c0-ae85f7c69561\") " pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:44.018532 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.018487    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvlj\" (UniqueName: \"kubernetes.io/projected/c58b4f94-071e-400c-bc9b-e72bb86a324f-kube-api-access-mjvlj\") pod \"multus-additional-cni-plugins-p7m7l\" (UID: \"c58b4f94-071e-400c-bc9b-e72bb86a324f\") " pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.018929 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.018907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fvrf\" (UniqueName: \"kubernetes.io/projected/f256da54-df5c-4dbc-9f2f-09b35844ea45-kube-api-access-6fvrf\") pod \"tuned-gbxrx\" (UID: \"f256da54-df5c-4dbc-9f2f-09b35844ea45\") " pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.019302 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.019213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m62q\" (UniqueName: \"kubernetes.io/projected/aba46778-09e3-41cb-a6dc-c26359982ef3-kube-api-access-6m62q\") pod \"aws-ebs-csi-driver-node-9nqwl\" (UID: \"aba46778-09e3-41cb-a6dc-c26359982ef3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.021624 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.021604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4ln\" (UniqueName: \"kubernetes.io/projected/0f8b5d92-d112-4bbc-a824-b4a96710d5db-kube-api-access-sg4ln\") pod \"ovnkube-node-8nfkh\" (UID: \"0f8b5d92-d112-4bbc-a824-b4a96710d5db\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.096935 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.096892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:52:44.111946 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.111719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vp5cx"
Sep 09 20:52:44.122985 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.122954    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m92df"
Sep 09 20:52:44.130678 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.130652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mxtw2"
Sep 09 20:52:44.139572 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.139548    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl"
Sep 09 20:52:44.147521 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.147493    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p7m7l"
Sep 09 20:52:44.159635 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.159611    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gbxrx"
Sep 09 20:52:44.170611 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.170586    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mmcfd"
Sep 09 20:52:44.179638 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.179612    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:52:44.412785 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.412752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:44.412785 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.412790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:44.413055 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.412813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:44.413055 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:44.412911    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:44.413055 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:44.412925    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:44.413055 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:44.412937    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:44.413055 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:44.412960    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:44.413055 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:44.412975    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gflfq for pod openshift-network-diagnostics/network-check-target-q2w92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:44.413055 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:44.412980    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs podName:d43fba51-6814-4c9e-a5c8-a1dd796b7386 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:45.412960074 +0000 UTC m=+4.037434024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs") pod "network-metrics-daemon-29hf4" (UID: "d43fba51-6814-4c9e-a5c8-a1dd796b7386") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:44.413055 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:44.413002    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret podName:2685e808-1a0b-4fb0-b19a-876ead4ad42e nodeName:}" failed. No retries permitted until 2026-09-09 20:52:45.412991994 +0000 UTC m=+4.037465918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret") pod "global-pull-secret-syncer-nrhbs" (UID: "2685e808-1a0b-4fb0-b19a-876ead4ad42e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:44.413055 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:44.413028    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq podName:f982e35b-5816-44a8-92a2-8b363ebe4c36 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:45.413010447 +0000 UTC m=+4.037484387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gflfq" (UniqueName: "kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq") pod "network-check-target-q2w92" (UID: "f982e35b-5816-44a8-92a2-8b363ebe4c36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:44.842246 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.842179    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:44 +0000 UTC" deadline="2028-03-04 09:18:23.619305862 +0000 UTC"
Sep 09 20:52:44.842246 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:44.842241    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12996h25m38.777068153s"
Sep 09 20:52:45.070579 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:45.070537    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad8f34d_a7d6_4549_ac85_b9f9e0589a5a.slice/crio-8285fff21b816d3ceb9d2616dd9db38e9f4ca0c6864b8e84cc6c2e3925cf2697 WatchSource:0}: Error finding container 8285fff21b816d3ceb9d2616dd9db38e9f4ca0c6864b8e84cc6c2e3925cf2697: Status 404 returned error can't find the container with id 8285fff21b816d3ceb9d2616dd9db38e9f4ca0c6864b8e84cc6c2e3925cf2697
Sep 09 20:52:45.071577 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:45.071543    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf256da54_df5c_4dbc_9f2f_09b35844ea45.slice/crio-b4c338c68af89c218d24aa1c2f41119d585fbc2bc2cb7b68872ccaf46df2e1c7 WatchSource:0}: Error finding container b4c338c68af89c218d24aa1c2f41119d585fbc2bc2cb7b68872ccaf46df2e1c7: Status 404 returned error can't find the container with id b4c338c68af89c218d24aa1c2f41119d585fbc2bc2cb7b68872ccaf46df2e1c7
Sep 09 20:52:45.072268 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:45.072204    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c2c926_9449_4539_b439_06fb5d910856.slice/crio-6204053255b1aadceab4d6921a2a8823d7c4643439d51152fa3ae94953c3ff8b WatchSource:0}: Error finding container 6204053255b1aadceab4d6921a2a8823d7c4643439d51152fa3ae94953c3ff8b: Status 404 returned error can't find the container with id 6204053255b1aadceab4d6921a2a8823d7c4643439d51152fa3ae94953c3ff8b
Sep 09 20:52:45.074186 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:45.073629    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02275490_d835_48ad_836f_4eb605bd4d0f.slice/crio-40902f96b986f7aea9ad569dc40c2f93ba70c1e3cb22c8bc5d4826b5b6013a4e WatchSource:0}: Error finding container 40902f96b986f7aea9ad569dc40c2f93ba70c1e3cb22c8bc5d4826b5b6013a4e: Status 404 returned error can't find the container with id 40902f96b986f7aea9ad569dc40c2f93ba70c1e3cb22c8bc5d4826b5b6013a4e
Sep 09 20:52:45.076929 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:45.075876    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c0bc3b_be18_47e4_bd35_d90287e22c4b.slice/crio-53d458af4435c8c5660ca3a3d07478bd0afb1c16e75afc818fbe4cea17ad33d3 WatchSource:0}: Error finding container 53d458af4435c8c5660ca3a3d07478bd0afb1c16e75afc818fbe4cea17ad33d3: Status 404 returned error can't find the container with id 53d458af4435c8c5660ca3a3d07478bd0afb1c16e75afc818fbe4cea17ad33d3
Sep 09 20:52:45.103137 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:45.103100    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8b5d92_d112_4bbc_a824_b4a96710d5db.slice/crio-af3c7fbd9d2d462feafc5ff80e6f6f77568e04c9ee0aa47269d7bff62ed1c01b WatchSource:0}: Error finding container af3c7fbd9d2d462feafc5ff80e6f6f77568e04c9ee0aa47269d7bff62ed1c01b: Status 404 returned error can't find the container with id af3c7fbd9d2d462feafc5ff80e6f6f77568e04c9ee0aa47269d7bff62ed1c01b
Sep 09 20:52:45.104745 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:52:45.104706    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba46778_09e3_41cb_a6dc_c26359982ef3.slice/crio-571023d7379e452e9c33255cecbe80d07ab3208a38fa69c7534e5dd3aab3e1d8 WatchSource:0}: Error finding container 571023d7379e452e9c33255cecbe80d07ab3208a38fa69c7534e5dd3aab3e1d8: Status 404 returned error can't find the container with id 571023d7379e452e9c33255cecbe80d07ab3208a38fa69c7534e5dd3aab3e1d8
Sep 09 20:52:45.420919 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.420738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:45.420989 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.420976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:45.421026 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.420999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:45.421075 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.420889    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:45.421117 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.421108    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs podName:d43fba51-6814-4c9e-a5c8-a1dd796b7386 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:47.4210952 +0000 UTC m=+6.045569120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs") pod "network-metrics-daemon-29hf4" (UID: "d43fba51-6814-4c9e-a5c8-a1dd796b7386") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:45.421410 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.421396    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:45.421450 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.421414    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:45.421450 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.421424    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gflfq for pod openshift-network-diagnostics/network-check-target-q2w92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:45.421511 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.421455    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq podName:f982e35b-5816-44a8-92a2-8b363ebe4c36 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:47.421445321 +0000 UTC m=+6.045919242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gflfq" (UniqueName: "kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq") pod "network-check-target-q2w92" (UID: "f982e35b-5816-44a8-92a2-8b363ebe4c36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:45.421511 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.421064    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:45.421511 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.421482    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret podName:2685e808-1a0b-4fb0-b19a-876ead4ad42e nodeName:}" failed. No retries permitted until 2026-09-09 20:52:47.421474771 +0000 UTC m=+6.045948691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret") pod "global-pull-secret-syncer-nrhbs" (UID: "2685e808-1a0b-4fb0-b19a-876ead4ad42e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:45.842659 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.842604    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:44 +0000 UTC" deadline="2028-02-26 00:53:27.94160307 +0000 UTC"
Sep 09 20:52:45.842659 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.842652    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12820h0m42.098956919s"
Sep 09 20:52:45.862276 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.862239    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:45.862459 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.862393    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:52:45.862827 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.862802    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:45.862923 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.862905    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:52:45.862997 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.862985    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:45.863091 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:45.863070    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:52:45.881994 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.881953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vp5cx" event={"ID":"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a","Type":"ContainerStarted","Data":"8285fff21b816d3ceb9d2616dd9db38e9f4ca0c6864b8e84cc6c2e3925cf2697"}
Sep 09 20:52:45.903614 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.903523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl" event={"ID":"aba46778-09e3-41cb-a6dc-c26359982ef3","Type":"ContainerStarted","Data":"571023d7379e452e9c33255cecbe80d07ab3208a38fa69c7534e5dd3aab3e1d8"}
Sep 09 20:52:45.910058 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.910014    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" event={"ID":"0f8b5d92-d112-4bbc-a824-b4a96710d5db","Type":"ContainerStarted","Data":"af3c7fbd9d2d462feafc5ff80e6f6f77568e04c9ee0aa47269d7bff62ed1c01b"}
Sep 09 20:52:45.921703 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.921660    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mxtw2" event={"ID":"c468bb92-a81e-4d78-88c0-ae85f7c69561","Type":"ContainerStarted","Data":"e62d72868cffa7fa1803291211de29cc6ae07b0f32699482f5cf6dce0e4c5c83"}
Sep 09 20:52:45.927568 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.927524    2583 generic.go:356] "Generic (PLEG): container finished" podID="83c08e0d52b4ee9262822078dda7d62a" containerID="5b0cbf8cc8e7250c4e29375e095e79d40c90cf5a7ae9f2dcae45fdb2d50f01d5" exitCode=0
Sep 09 20:52:45.927716 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.927675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" event={"ID":"83c08e0d52b4ee9262822078dda7d62a","Type":"ContainerDied","Data":"5b0cbf8cc8e7250c4e29375e095e79d40c90cf5a7ae9f2dcae45fdb2d50f01d5"}
Sep 09 20:52:45.939082 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.939053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal" event={"ID":"9fdee4d29f0b1359d09bd37b4c77420b","Type":"ContainerStarted","Data":"47c858bccae55e6c12d0a37304b6207a620b884f4c72773271ec7a2bc881a931"}
Sep 09 20:52:45.949464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.949289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7m7l" event={"ID":"c58b4f94-071e-400c-bc9b-e72bb86a324f","Type":"ContainerStarted","Data":"7e0bb30d9163bf26f6f9b20ad1abbb8dcc59c48707ddebcd85c641254e1dc967"}
Sep 09 20:52:45.966410 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.966371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6zp7" event={"ID":"62c0bc3b-be18-47e4-bd35-d90287e22c4b","Type":"ContainerStarted","Data":"53d458af4435c8c5660ca3a3d07478bd0afb1c16e75afc818fbe4cea17ad33d3"}
Sep 09 20:52:45.970550 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.970515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mmcfd" event={"ID":"b5c2c926-9449-4539-b439-06fb5d910856","Type":"ContainerStarted","Data":"6204053255b1aadceab4d6921a2a8823d7c4643439d51152fa3ae94953c3ff8b"}
Sep 09 20:52:45.988409 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.988326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m92df" event={"ID":"02275490-d835-48ad-836f-4eb605bd4d0f","Type":"ContainerStarted","Data":"40902f96b986f7aea9ad569dc40c2f93ba70c1e3cb22c8bc5d4826b5b6013a4e"}
Sep 09 20:52:45.993008 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:45.992978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gbxrx" event={"ID":"f256da54-df5c-4dbc-9f2f-09b35844ea45","Type":"ContainerStarted","Data":"b4c338c68af89c218d24aa1c2f41119d585fbc2bc2cb7b68872ccaf46df2e1c7"}
Sep 09 20:52:47.005366 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:47.005322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" event={"ID":"83c08e0d52b4ee9262822078dda7d62a","Type":"ContainerStarted","Data":"cf09377895f74dade96f89f10e35196432c58635d5e38dc9fc950f144581f34e"}
Sep 09 20:52:47.039889 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:47.039813    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal" podStartSLOduration=4.039789539 podStartE2EDuration="4.039789539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:45.997520115 +0000 UTC m=+4.621994051" watchObservedRunningTime="2026-09-09 20:52:47.039789539 +0000 UTC m=+5.664263486"
Sep 09 20:52:47.438053 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:47.438002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:47.438273 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:47.438132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:47.438273 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:47.438162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:47.438384 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.438283    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:47.438384 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.438343    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret podName:2685e808-1a0b-4fb0-b19a-876ead4ad42e nodeName:}" failed. No retries permitted until 2026-09-09 20:52:51.438325753 +0000 UTC m=+10.062799674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret") pod "global-pull-secret-syncer-nrhbs" (UID: "2685e808-1a0b-4fb0-b19a-876ead4ad42e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:47.438944 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.438728    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:47.438944 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.438777    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs podName:d43fba51-6814-4c9e-a5c8-a1dd796b7386 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:51.438762373 +0000 UTC m=+10.063236299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs") pod "network-metrics-daemon-29hf4" (UID: "d43fba51-6814-4c9e-a5c8-a1dd796b7386") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:47.438944 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.438852    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:47.438944 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.438865    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:47.438944 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.438877    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gflfq for pod openshift-network-diagnostics/network-check-target-q2w92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:47.438944 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.438911    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq podName:f982e35b-5816-44a8-92a2-8b363ebe4c36 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:51.438900171 +0000 UTC m=+10.063374092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gflfq" (UniqueName: "kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq") pod "network-check-target-q2w92" (UID: "f982e35b-5816-44a8-92a2-8b363ebe4c36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:47.860261 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:47.860229    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:47.860444 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:47.860276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:47.860444 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:47.860229    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:47.860444 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.860380    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:52:47.860621 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.860471    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:52:47.860621 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:47.860550    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:52:49.860378 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:49.860343    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:49.860833 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:49.860393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:49.860833 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:49.860393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:49.860833 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:49.860482    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:52:49.860833 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:49.860581    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:52:49.860833 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:49.860660    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:52:51.475898 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:51.475866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:51.476381 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:51.475904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:51.476381 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:51.475937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:51.476381 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.476054    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:51.476381 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.476136    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs podName:d43fba51-6814-4c9e-a5c8-a1dd796b7386 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.476094798 +0000 UTC m=+18.100568737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs") pod "network-metrics-daemon-29hf4" (UID: "d43fba51-6814-4c9e-a5c8-a1dd796b7386") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:51.476767 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.476580    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:51.476767 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.476604    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:51.476767 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.476616    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gflfq for pod openshift-network-diagnostics/network-check-target-q2w92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:51.476767 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.476662    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq podName:f982e35b-5816-44a8-92a2-8b363ebe4c36 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.476650261 +0000 UTC m=+18.101124179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gflfq" (UniqueName: "kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq") pod "network-check-target-q2w92" (UID: "f982e35b-5816-44a8-92a2-8b363ebe4c36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:51.476767 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.476717    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:51.476767 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.476743    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret podName:2685e808-1a0b-4fb0-b19a-876ead4ad42e nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.476736815 +0000 UTC m=+18.101210733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret") pod "global-pull-secret-syncer-nrhbs" (UID: "2685e808-1a0b-4fb0-b19a-876ead4ad42e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:51.860312 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:51.860274    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:51.860492 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.860401    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:52:51.860492 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:51.860474    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:51.860579 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:51.860568    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:51.860634 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.860606    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:52:51.860680 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:51.860647    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:52:53.860154 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:53.859590    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:53.860154 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:53.859590    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:53.860154 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:53.859769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:53.860154 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:53.859764    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:52:53.860154 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:53.859868    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:52:53.860154 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:53.859949    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:52:55.859988 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:55.859950    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:55.860420 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:55.859953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:55.860420 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:55.860057    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:52:55.860420 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:55.860141    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:52:55.860420 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:55.859961    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:55.860420 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:55.860212    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:52:57.859803 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:57.859347    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:57.859803 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:57.859354    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:57.859803 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:57.859465    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:52:57.859803 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:57.859484    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:57.859803 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:57.859604    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:52:57.859803 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:57.859690    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:52:59.534999 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:59.534953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:59.534999 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:59.535004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:59.535566 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:59.535038    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:59.535566 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.535162    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:59.535566 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.535162    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:59.535566 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.535199    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:59.535566 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.535211    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gflfq for pod openshift-network-diagnostics/network-check-target-q2w92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:59.535566 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.535225    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs podName:d43fba51-6814-4c9e-a5c8-a1dd796b7386 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:15.535203674 +0000 UTC m=+34.159677606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs") pod "network-metrics-daemon-29hf4" (UID: "d43fba51-6814-4c9e-a5c8-a1dd796b7386") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:59.535566 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.535258    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq podName:f982e35b-5816-44a8-92a2-8b363ebe4c36 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:15.535242016 +0000 UTC m=+34.159715937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gflfq" (UniqueName: "kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq") pod "network-check-target-q2w92" (UID: "f982e35b-5816-44a8-92a2-8b363ebe4c36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:59.535566 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.535172    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:59.535566 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.535337    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret podName:2685e808-1a0b-4fb0-b19a-876ead4ad42e nodeName:}" failed. No retries permitted until 2026-09-09 20:53:15.535317341 +0000 UTC m=+34.159791277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret") pod "global-pull-secret-syncer-nrhbs" (UID: "2685e808-1a0b-4fb0-b19a-876ead4ad42e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:59.859956 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:59.859921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:52:59.860142 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:59.859921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:52:59.860142 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.860065    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:52:59.860277 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.860157    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:52:59.860277 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:52:59.859921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:52:59.860277 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:52:59.860253    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:01.860849 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:01.860809    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:01.861341 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:01.860912    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:53:01.861341 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:01.860920    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:01.861341 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:01.860944    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:01.861341 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:01.860989    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:53:01.861341 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:01.861095    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:03.860382 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:03.860047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:03.861185 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:03.860046    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:03.861185 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:03.860470    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:53:03.861185 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:03.860096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:03.861185 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:03.860585    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:03.861185 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:03.860702    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:53:04.043025 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.042374    2583 generic.go:356] "Generic (PLEG): container finished" podID="c58b4f94-071e-400c-bc9b-e72bb86a324f" containerID="b5d233905a228fe23c23d3779bd24c1234b0ce2b12c9a9b1f6ffbf57ec180329" exitCode=0
Sep 09 20:53:04.043025 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.042461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7m7l" event={"ID":"c58b4f94-071e-400c-bc9b-e72bb86a324f","Type":"ContainerDied","Data":"b5d233905a228fe23c23d3779bd24c1234b0ce2b12c9a9b1f6ffbf57ec180329"}
Sep 09 20:53:04.044486 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.044458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6zp7" event={"ID":"62c0bc3b-be18-47e4-bd35-d90287e22c4b","Type":"ContainerStarted","Data":"49f5be05c24e1332247b37537b09d11aec1fd0e369d901b175810c9b4fbaa82e"}
Sep 09 20:53:04.046346 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.046314    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mmcfd" event={"ID":"b5c2c926-9449-4539-b439-06fb5d910856","Type":"ContainerStarted","Data":"172617311d75ac3b1e6d951d78096ae3b486870ef4fba40eddfa13d576f359ad"}
Sep 09 20:53:04.048333 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.048309    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m92df" event={"ID":"02275490-d835-48ad-836f-4eb605bd4d0f","Type":"ContainerStarted","Data":"2513449961abb58d14af0988510bbdfdc76bbc3772e729732c1b95eb442841b5"}
Sep 09 20:53:04.050061 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.050039    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gbxrx" event={"ID":"f256da54-df5c-4dbc-9f2f-09b35844ea45","Type":"ContainerStarted","Data":"9bc8a90061b5915cb276b22d61682d9f41bb6c3abbe305f14af75d29d3bed8eb"}
Sep 09 20:53:04.051646 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.051626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl" event={"ID":"aba46778-09e3-41cb-a6dc-c26359982ef3","Type":"ContainerStarted","Data":"e3a33b9d0fcfe672fdec872e60b6c6e4ea767904c28383ae77cd6b8ecb9eea47"}
Sep 09 20:53:04.054992 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.054969    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" event={"ID":"0f8b5d92-d112-4bbc-a824-b4a96710d5db","Type":"ContainerStarted","Data":"58c1f4129970b7ba4f07691d2da15a0bc26b1ac9430c850b30ea788211610370"}
Sep 09 20:53:04.055089 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.055048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" event={"ID":"0f8b5d92-d112-4bbc-a824-b4a96710d5db","Type":"ContainerStarted","Data":"c22f6b1ba74c06a159f774eb833fb18226f883ace873aa7147443e6bd3422243"}
Sep 09 20:53:04.055168 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.055097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" event={"ID":"0f8b5d92-d112-4bbc-a824-b4a96710d5db","Type":"ContainerStarted","Data":"f78148ab99874e36ba1928e546507fca79b8cccffdaa1d1372592386ec90fe22"}
Sep 09 20:53:04.055214 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.055173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" event={"ID":"0f8b5d92-d112-4bbc-a824-b4a96710d5db","Type":"ContainerStarted","Data":"44ec5a72ec4ecdd5642ac77f5521ca3521c0b10b0b31e0cf92277b1465588066"}
Sep 09 20:53:04.055254 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.055220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" event={"ID":"0f8b5d92-d112-4bbc-a824-b4a96710d5db","Type":"ContainerStarted","Data":"3fcdd7d519d65514c50d5479a162a27b4f321cf48148ba702e4bff4899c58a92"}
Sep 09 20:53:04.055300 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.055268    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" event={"ID":"0f8b5d92-d112-4bbc-a824-b4a96710d5db","Type":"ContainerStarted","Data":"5279a7818ac4e96ab6307c0863f365b0b59b178d1dd6db39154ab940d347d9b7"}
Sep 09 20:53:04.057009 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.056587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mxtw2" event={"ID":"c468bb92-a81e-4d78-88c0-ae85f7c69561","Type":"ContainerStarted","Data":"7c4404b345e336f805b8c19fb1bc356f46336c8dc6aa394afa5395427c52a0c4"}
Sep 09 20:53:04.081881 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.081816    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" podStartSLOduration=21.081803258 podStartE2EDuration="21.081803258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:47.040849009 +0000 UTC m=+5.665322954" watchObservedRunningTime="2026-09-09 20:53:04.081803258 +0000 UTC m=+22.706277197"
Sep 09 20:53:04.103034 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.102987    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k6zp7" podStartSLOduration=4.049145282 podStartE2EDuration="22.102973824s" totalImagesPullingTime="18.053828542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:04.102757146 +0000 UTC m=+22.727231087" watchObservedRunningTime="2026-09-09 20:53:04.102973824 +0000 UTC m=+22.727447761"
Sep 09 20:53:04.122532 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.122484    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m92df" podStartSLOduration=3.972907198 podStartE2EDuration="22.122466563s" totalImagesPullingTime="18.149559365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:04.122164192 +0000 UTC m=+22.746638132" watchObservedRunningTime="2026-09-09 20:53:04.122466563 +0000 UTC m=+22.746940504"
Sep 09 20:53:04.142464 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.142399    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gbxrx" podStartSLOduration=4.045097275 podStartE2EDuration="22.142379293s" totalImagesPullingTime="18.097282018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:04.141438577 +0000 UTC m=+22.765912517" watchObservedRunningTime="2026-09-09 20:53:04.142379293 +0000 UTC m=+22.766853234"
Sep 09 20:53:04.160176 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.160123    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mmcfd" podStartSLOduration=4.067388502 podStartE2EDuration="22.160088979s" totalImagesPullingTime="18.092700477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:04.159722282 +0000 UTC m=+22.784196223" watchObservedRunningTime="2026-09-09 20:53:04.160088979 +0000 UTC m=+22.784562916"
Sep 09 20:53:04.180395 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.180341    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mxtw2" podStartSLOduration=4.126366937 podStartE2EDuration="22.180325052s" totalImagesPullingTime="18.053958115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:04.180316407 +0000 UTC m=+22.804790348" watchObservedRunningTime="2026-09-09 20:53:04.180325052 +0000 UTC m=+22.804798993"
Sep 09 20:53:04.832015 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.831986    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:53:04.865145 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.865035    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:53:04.832005565Z","UUID":"14420921-4567-4ce3-89ba-b4a488217bb8","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:53:04.866560 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.866535    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:53:04.866676 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.866568    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:53:04.950407 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.950372    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:53:04.950932 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:04.950913    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:53:05.061078 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:05.061036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vp5cx" event={"ID":"dad8f34d-a7d6-4549-ac85-b9f9e0589a5a","Type":"ContainerStarted","Data":"3cfcbd2ab760b84012869ccbc5a1c86dad2dcc962d59c802cb357f25cf831b84"}
Sep 09 20:53:05.063402 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:05.063372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl" event={"ID":"aba46778-09e3-41cb-a6dc-c26359982ef3","Type":"ContainerStarted","Data":"1c29fed7009e2e576b226552995acf90325713e7b54d1d139dba2eeed6dd08ec"}
Sep 09 20:53:05.064060 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:05.064037    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:53:05.064182 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:05.064099    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k6zp7"
Sep 09 20:53:05.098910 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:05.098813    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vp5cx" podStartSLOduration=5.003000106 podStartE2EDuration="23.098797984s" totalImagesPullingTime="18.095797878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:05.081366861 +0000 UTC m=+23.705840803" watchObservedRunningTime="2026-09-09 20:53:05.098797984 +0000 UTC m=+23.723271924"
Sep 09 20:53:05.859702 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:05.859669    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:05.859858 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:05.859670    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:05.859858 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:05.859670    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:05.859858 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:05.859809    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:53:05.860010 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:05.859902    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:05.860010 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:05.859992    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:53:06.070532 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:06.070147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" event={"ID":"0f8b5d92-d112-4bbc-a824-b4a96710d5db","Type":"ContainerStarted","Data":"ad1883757f67609f3386a2e1f105d283a7ed83e3afbf64792adbeb00d1dd6169"}
Sep 09 20:53:07.075862 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:07.075815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl" event={"ID":"aba46778-09e3-41cb-a6dc-c26359982ef3","Type":"ContainerStarted","Data":"fddbecf39299e4ce78e52f5ff6c6c425692752f2a84c696658a54d5a4a03755c"}
Sep 09 20:53:07.100910 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:07.100847    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nqwl" podStartSLOduration=3.896008921 podStartE2EDuration="25.100824789s" totalImagesPullingTime="21.204815868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:07.099338856 +0000 UTC m=+25.723812800" watchObservedRunningTime="2026-09-09 20:53:07.100824789 +0000 UTC m=+25.725298733"
Sep 09 20:53:07.859243 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:07.859201    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:07.859453 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:07.859215    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:07.859453 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:07.859332    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:53:07.859453 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:07.859368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:07.859626 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:07.859464    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:53:07.859626 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:07.859558    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:09.083841 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.083533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" event={"ID":"0f8b5d92-d112-4bbc-a824-b4a96710d5db","Type":"ContainerStarted","Data":"d22d426c86d488907ccadd7ef8e651d97da69273413c0eb525b5f3be59505088"}
Sep 09 20:53:09.084502 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.083843    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:53:09.084502 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.083897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:53:09.084502 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.083941    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:53:09.085823 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.085782    2583 generic.go:356] "Generic (PLEG): container finished" podID="c58b4f94-071e-400c-bc9b-e72bb86a324f" containerID="d364d6c9a8c3a77589e7d9d041d81a3e838a5814f90a058f5715b289b20817dd" exitCode=0
Sep 09 20:53:09.085956 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.085826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7m7l" event={"ID":"c58b4f94-071e-400c-bc9b-e72bb86a324f","Type":"ContainerDied","Data":"d364d6c9a8c3a77589e7d9d041d81a3e838a5814f90a058f5715b289b20817dd"}
Sep 09 20:53:09.099609 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.099584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:53:09.099711 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.099689    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:53:09.125666 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.123443    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh" podStartSLOduration=9.006586364 podStartE2EDuration="27.123424278s" totalImagesPullingTime="18.116837914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:09.121834719 +0000 UTC m=+27.746308659" watchObservedRunningTime="2026-09-09 20:53:09.123424278 +0000 UTC m=+27.747898225"
Sep 09 20:53:09.859832 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.859793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:09.859832 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.859834    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:09.860152 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:09.859947    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:53:09.860152 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:09.860018    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:53:09.860152 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:09.860078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:09.860311 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:09.860167    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:10.674064 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:10.673835    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29hf4"]
Sep 09 20:53:10.674457 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:10.674135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:10.674457 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:10.674226    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:53:10.677991 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:10.677962    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q2w92"]
Sep 09 20:53:10.678173 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:10.678053    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:10.678250 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:10.678182    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:10.678635 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:10.678615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nrhbs"]
Sep 09 20:53:10.678696 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:10.678690    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:10.678774 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:10.678759    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:53:11.092324 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:11.092289    2583 generic.go:356] "Generic (PLEG): container finished" podID="c58b4f94-071e-400c-bc9b-e72bb86a324f" containerID="c854f3cc7917110dd7ae2bf737022628b53ea1b3f86572343dd103a715ace7f5" exitCode=0
Sep 09 20:53:11.092485 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:11.092370    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7m7l" event={"ID":"c58b4f94-071e-400c-bc9b-e72bb86a324f","Type":"ContainerDied","Data":"c854f3cc7917110dd7ae2bf737022628b53ea1b3f86572343dd103a715ace7f5"}
Sep 09 20:53:11.860004 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:11.859971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:11.860486 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:11.860066    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:12.859977 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:12.859791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:12.860164 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:12.859823    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:12.860164 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:12.860063    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:53:12.860164 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:12.860155    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:53:13.099146 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:13.099096    2583 generic.go:356] "Generic (PLEG): container finished" podID="c58b4f94-071e-400c-bc9b-e72bb86a324f" containerID="b001a5583051c0e802656e6ddce8b14afaa29630430f285f10d8854e02093244" exitCode=0
Sep 09 20:53:13.099303 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:13.099169    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7m7l" event={"ID":"c58b4f94-071e-400c-bc9b-e72bb86a324f","Type":"ContainerDied","Data":"b001a5583051c0e802656e6ddce8b14afaa29630430f285f10d8854e02093244"}
Sep 09 20:53:13.860059 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:13.860025    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:13.860240 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:13.860141    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:14.860126 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:14.860077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:14.860324 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:14.860077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:14.860324 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:14.860219    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nrhbs" podUID="2685e808-1a0b-4fb0-b19a-876ead4ad42e"
Sep 09 20:53:14.860324 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:14.860300    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29hf4" podUID="d43fba51-6814-4c9e-a5c8-a1dd796b7386"
Sep 09 20:53:15.556483 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:15.556437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:15.556483 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:15.556482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:15.556694 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:15.556504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:15.556694 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:15.556599    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:15.556694 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:15.556601    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:15.556694 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:15.556620    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:15.556694 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:15.556642    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:15.556694 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:15.556654    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gflfq for pod openshift-network-diagnostics/network-check-target-q2w92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:15.556694 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:15.556655    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs podName:d43fba51-6814-4c9e-a5c8-a1dd796b7386 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:47.556640311 +0000 UTC m=+66.181114230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs") pod "network-metrics-daemon-29hf4" (UID: "d43fba51-6814-4c9e-a5c8-a1dd796b7386") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:15.556694 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:15.556698    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret podName:2685e808-1a0b-4fb0-b19a-876ead4ad42e nodeName:}" failed. No retries permitted until 2026-09-09 20:53:47.556685551 +0000 UTC m=+66.181159470 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret") pod "global-pull-secret-syncer-nrhbs" (UID: "2685e808-1a0b-4fb0-b19a-876ead4ad42e") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:15.556934 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:15.556713    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq podName:f982e35b-5816-44a8-92a2-8b363ebe4c36 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:47.556705035 +0000 UTC m=+66.181178958 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gflfq" (UniqueName: "kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq") pod "network-check-target-q2w92" (UID: "f982e35b-5816-44a8-92a2-8b363ebe4c36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:15.859491 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:15.859404    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:15.859652 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:15.859543    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2w92" podUID="f982e35b-5816-44a8-92a2-8b363ebe4c36"
Sep 09 20:53:16.189298 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.189215    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeReady"
Sep 09 20:53:16.189733 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.189376    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:53:16.242905 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.242870    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rjl4w"]
Sep 09 20:53:16.247220 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.247190    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7plsw"]
Sep 09 20:53:16.247372 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.247306    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.249685 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.249659    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:53:16.249818 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.249751    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nr885\""
Sep 09 20:53:16.251160 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.251137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7plsw"
Sep 09 20:53:16.254433 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.254185    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:53:16.254433 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.254369    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:53:16.254602 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.254437    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:53:16.254602 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.254582    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7nzq\""
Sep 09 20:53:16.254699 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.254682    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:53:16.259593 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.259569    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rjl4w"]
Sep 09 20:53:16.262356 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.262335    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7plsw"]
Sep 09 20:53:16.362387 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.362352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-tmp-dir\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.362387 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.362397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-config-volume\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.362625 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.362431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/393acb6a-39f1-46c0-9c3a-29f50af68eff-cert\") pod \"ingress-canary-7plsw\" (UID: \"393acb6a-39f1-46c0-9c3a-29f50af68eff\") " pod="openshift-ingress-canary/ingress-canary-7plsw"
Sep 09 20:53:16.362625 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.362482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dh5\" (UniqueName: \"kubernetes.io/projected/393acb6a-39f1-46c0-9c3a-29f50af68eff-kube-api-access-62dh5\") pod \"ingress-canary-7plsw\" (UID: \"393acb6a-39f1-46c0-9c3a-29f50af68eff\") " pod="openshift-ingress-canary/ingress-canary-7plsw"
Sep 09 20:53:16.362625 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.362506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.362625 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.362545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8t5n\" (UniqueName: \"kubernetes.io/projected/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-kube-api-access-h8t5n\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.463754 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.463667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8t5n\" (UniqueName: \"kubernetes.io/projected/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-kube-api-access-h8t5n\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.463754 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.463736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-tmp-dir\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.463966 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.463770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-config-volume\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.463966 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.463803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/393acb6a-39f1-46c0-9c3a-29f50af68eff-cert\") pod \"ingress-canary-7plsw\" (UID: \"393acb6a-39f1-46c0-9c3a-29f50af68eff\") " pod="openshift-ingress-canary/ingress-canary-7plsw"
Sep 09 20:53:16.463966 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.463857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dh5\" (UniqueName: \"kubernetes.io/projected/393acb6a-39f1-46c0-9c3a-29f50af68eff-kube-api-access-62dh5\") pod \"ingress-canary-7plsw\" (UID: \"393acb6a-39f1-46c0-9c3a-29f50af68eff\") " pod="openshift-ingress-canary/ingress-canary-7plsw"
Sep 09 20:53:16.463966 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.463878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.464176 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:16.463990    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:16.464176 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:16.464046    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls podName:17fc2904-8ec6-4d26-8f39-14abdbe1a1cc nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.964030657 +0000 UTC m=+35.588504589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls") pod "dns-default-rjl4w" (UID: "17fc2904-8ec6-4d26-8f39-14abdbe1a1cc") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:16.464285 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.464196    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-tmp-dir\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.464617 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.464594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-config-volume\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.469656 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.469625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/393acb6a-39f1-46c0-9c3a-29f50af68eff-cert\") pod \"ingress-canary-7plsw\" (UID: \"393acb6a-39f1-46c0-9c3a-29f50af68eff\") " pod="openshift-ingress-canary/ingress-canary-7plsw"
Sep 09 20:53:16.475409 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.475383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dh5\" (UniqueName: \"kubernetes.io/projected/393acb6a-39f1-46c0-9c3a-29f50af68eff-kube-api-access-62dh5\") pod \"ingress-canary-7plsw\" (UID: \"393acb6a-39f1-46c0-9c3a-29f50af68eff\") " pod="openshift-ingress-canary/ingress-canary-7plsw"
Sep 09 20:53:16.476433 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.476407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8t5n\" (UniqueName: \"kubernetes.io/projected/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-kube-api-access-h8t5n\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.568680 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.568634    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7plsw"
Sep 09 20:53:16.859665 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.859630    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:16.859835 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.859805    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:16.868186 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.867797    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:16.868186 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.867840    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:16.868186 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.868147    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2slq\""
Sep 09 20:53:16.967313 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:16.967255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:16.967486 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:16.967428    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:16.967534 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:16.967509    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls podName:17fc2904-8ec6-4d26-8f39-14abdbe1a1cc nodeName:}" failed. No retries permitted until 2026-09-09 20:53:17.967490018 +0000 UTC m=+36.591963954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls") pod "dns-default-rjl4w" (UID: "17fc2904-8ec6-4d26-8f39-14abdbe1a1cc") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:17.859404 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:17.859370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:17.864432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:17.864191    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:17.864432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:17.864197    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:17.866016 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:17.865990    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njm8p\""
Sep 09 20:53:17.974146 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:17.973910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:17.974270 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:17.974051    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:17.974270 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:17.974245    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls podName:17fc2904-8ec6-4d26-8f39-14abdbe1a1cc nodeName:}" failed. No retries permitted until 2026-09-09 20:53:19.974230502 +0000 UTC m=+38.598704437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls") pod "dns-default-rjl4w" (UID: "17fc2904-8ec6-4d26-8f39-14abdbe1a1cc") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:18.082788 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:18.082761    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7plsw"]
Sep 09 20:53:18.111069 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:53:18.111038    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393acb6a_39f1_46c0_9c3a_29f50af68eff.slice/crio-1adf47206f88ce0c7102dd1e8a11a543d298998a522934f443dbdb3066673ee3 WatchSource:0}: Error finding container 1adf47206f88ce0c7102dd1e8a11a543d298998a522934f443dbdb3066673ee3: Status 404 returned error can't find the container with id 1adf47206f88ce0c7102dd1e8a11a543d298998a522934f443dbdb3066673ee3
Sep 09 20:53:18.115151 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:18.115127    2583 generic.go:356] "Generic (PLEG): container finished" podID="c58b4f94-071e-400c-bc9b-e72bb86a324f" containerID="b0a659fa0a6f38197a70cffc5cf566751cac72f0960922aaac21fbe517b5e584" exitCode=0
Sep 09 20:53:18.115254 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:18.115165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7m7l" event={"ID":"c58b4f94-071e-400c-bc9b-e72bb86a324f","Type":"ContainerDied","Data":"b0a659fa0a6f38197a70cffc5cf566751cac72f0960922aaac21fbe517b5e584"}
Sep 09 20:53:19.118790 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:19.118759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7plsw" event={"ID":"393acb6a-39f1-46c0-9c3a-29f50af68eff","Type":"ContainerStarted","Data":"1adf47206f88ce0c7102dd1e8a11a543d298998a522934f443dbdb3066673ee3"}
Sep 09 20:53:19.122383 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:19.122357    2583 generic.go:356] "Generic (PLEG): container finished" podID="c58b4f94-071e-400c-bc9b-e72bb86a324f" containerID="48d2cba2093f9ac758da01a32c24dcd4c240a54b1a0a223e8d5ab141efa80483" exitCode=0
Sep 09 20:53:19.122500 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:19.122415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7m7l" event={"ID":"c58b4f94-071e-400c-bc9b-e72bb86a324f","Type":"ContainerDied","Data":"48d2cba2093f9ac758da01a32c24dcd4c240a54b1a0a223e8d5ab141efa80483"}
Sep 09 20:53:19.991151 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:19.991084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:19.991351 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:19.991258    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:19.991351 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:19.991342    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls podName:17fc2904-8ec6-4d26-8f39-14abdbe1a1cc nodeName:}" failed. No retries permitted until 2026-09-09 20:53:23.991317604 +0000 UTC m=+42.615791546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls") pod "dns-default-rjl4w" (UID: "17fc2904-8ec6-4d26-8f39-14abdbe1a1cc") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:20.129083 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:20.129044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p7m7l" event={"ID":"c58b4f94-071e-400c-bc9b-e72bb86a324f","Type":"ContainerStarted","Data":"7d66a73521c97f01aaa5a1999b7277c5283968307d4cbdf5b8876fe2a30c69b0"}
Sep 09 20:53:20.158657 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:20.158594    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p7m7l" podStartSLOduration=8.259815962 podStartE2EDuration="38.15857129s" totalImagesPullingTime="29.898755328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:20.15748343 +0000 UTC m=+38.781957387" watchObservedRunningTime="2026-09-09 20:53:20.15857129 +0000 UTC m=+38.783045237"
Sep 09 20:53:21.133623 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:21.133393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7plsw" event={"ID":"393acb6a-39f1-46c0-9c3a-29f50af68eff","Type":"ContainerStarted","Data":"9a21dea0ca65e74095bf7db353330234d7acac6f7b5387938e3ea9edd972e726"}
Sep 09 20:53:21.152360 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:21.152231    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7plsw" podStartSLOduration=2.47774102 podStartE2EDuration="5.152213005s" totalImagesPullingTime="2.674471985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:21.151428094 +0000 UTC m=+39.775902034" watchObservedRunningTime="2026-09-09 20:53:21.152213005 +0000 UTC m=+39.776686946"
Sep 09 20:53:24.019873 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:24.019831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:24.020254 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:24.019957    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:24.020254 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:24.020009    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls podName:17fc2904-8ec6-4d26-8f39-14abdbe1a1cc nodeName:}" failed. No retries permitted until 2026-09-09 20:53:32.019995086 +0000 UTC m=+50.644469004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls") pod "dns-default-rjl4w" (UID: "17fc2904-8ec6-4d26-8f39-14abdbe1a1cc") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:32.074331 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:32.074285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:32.074742 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:32.074432    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:32.074742 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:32.074504    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls podName:17fc2904-8ec6-4d26-8f39-14abdbe1a1cc nodeName:}" failed. No retries permitted until 2026-09-09 20:53:48.074488074 +0000 UTC m=+66.698961993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls") pod "dns-default-rjl4w" (UID: "17fc2904-8ec6-4d26-8f39-14abdbe1a1cc") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:41.112739 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:41.112705    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nfkh"
Sep 09 20:53:47.583455 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.583416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:47.583455 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.583459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:47.583987 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.583484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:53:47.587507 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.587481    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:47.587605 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.587532    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:47.587738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.587715    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:47.594188 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:47.594169    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:53:47.594283 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:47.594236    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs podName:d43fba51-6814-4c9e-a5c8-a1dd796b7386 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:51.5942145 +0000 UTC m=+130.218688419 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs") pod "network-metrics-daemon-29hf4" (UID: "d43fba51-6814-4c9e-a5c8-a1dd796b7386") : secret "metrics-daemon-secret" not found
Sep 09 20:53:47.598074 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.598056    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2685e808-1a0b-4fb0-b19a-876ead4ad42e-original-pull-secret\") pod \"global-pull-secret-syncer-nrhbs\" (UID: \"2685e808-1a0b-4fb0-b19a-876ead4ad42e\") " pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:47.598896 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.598877    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:47.606895 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.606874    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gflfq\" (UniqueName: \"kubernetes.io/projected/f982e35b-5816-44a8-92a2-8b363ebe4c36-kube-api-access-gflfq\") pod \"network-check-target-q2w92\" (UID: \"f982e35b-5816-44a8-92a2-8b363ebe4c36\") " pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:47.771227 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.771184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nrhbs"
Sep 09 20:53:47.876415 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.876387    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njm8p\""
Sep 09 20:53:47.882491 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:47.882470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:48.077597 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:48.077565    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nrhbs"]
Sep 09 20:53:48.080980 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:53:48.080943    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2685e808_1a0b_4fb0_b19a_876ead4ad42e.slice/crio-02a58f93a933f1f3d0ac1ad1a202d052f7710eba5197976942ac300c9ba17004 WatchSource:0}: Error finding container 02a58f93a933f1f3d0ac1ad1a202d052f7710eba5197976942ac300c9ba17004: Status 404 returned error can't find the container with id 02a58f93a933f1f3d0ac1ad1a202d052f7710eba5197976942ac300c9ba17004
Sep 09 20:53:48.087390 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:48.087369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:53:48.087515 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:48.087503    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:48.087575 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:53:48.087565    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls podName:17fc2904-8ec6-4d26-8f39-14abdbe1a1cc nodeName:}" failed. No retries permitted until 2026-09-09 20:54:20.087551371 +0000 UTC m=+98.712025290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls") pod "dns-default-rjl4w" (UID: "17fc2904-8ec6-4d26-8f39-14abdbe1a1cc") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:48.176713 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:48.176628    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q2w92"]
Sep 09 20:53:48.180099 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:53:48.180077    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf982e35b_5816_44a8_92a2_8b363ebe4c36.slice/crio-c66594a6f482cf82f3ca0ccb0588ae99018648dc474467e0de56e659ea059f2b WatchSource:0}: Error finding container c66594a6f482cf82f3ca0ccb0588ae99018648dc474467e0de56e659ea059f2b: Status 404 returned error can't find the container with id c66594a6f482cf82f3ca0ccb0588ae99018648dc474467e0de56e659ea059f2b
Sep 09 20:53:48.200203 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:48.200170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q2w92" event={"ID":"f982e35b-5816-44a8-92a2-8b363ebe4c36","Type":"ContainerStarted","Data":"c66594a6f482cf82f3ca0ccb0588ae99018648dc474467e0de56e659ea059f2b"}
Sep 09 20:53:48.201315 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:48.201290    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nrhbs" event={"ID":"2685e808-1a0b-4fb0-b19a-876ead4ad42e","Type":"ContainerStarted","Data":"02a58f93a933f1f3d0ac1ad1a202d052f7710eba5197976942ac300c9ba17004"}
Sep 09 20:53:53.214451 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:53.214415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q2w92" event={"ID":"f982e35b-5816-44a8-92a2-8b363ebe4c36","Type":"ContainerStarted","Data":"38540c8d2c9ef8001d126bbd3d76713c56a3cca19574f2a1bd03192a92173204"}
Sep 09 20:53:53.214993 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:53.214547    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:53:53.215816 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:53.215794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nrhbs" event={"ID":"2685e808-1a0b-4fb0-b19a-876ead4ad42e","Type":"ContainerStarted","Data":"cc6146cc10c425558bd62ebc50b546d8ce3219fe6ca1baf94b2891ebb37db93a"}
Sep 09 20:53:53.235681 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:53.235632    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q2w92" podStartSLOduration=66.801058106 podStartE2EDuration="1m11.235616455s" totalImagesPullingTime="4.434558349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:53.234296757 +0000 UTC m=+71.858770698" watchObservedRunningTime="2026-09-09 20:53:53.235616455 +0000 UTC m=+71.860090396"
Sep 09 20:53:53.251094 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:53:53.251041    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nrhbs" podStartSLOduration=66.711862079 podStartE2EDuration="1m11.251025964s" totalImagesPullingTime="4.539163885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:53.250371564 +0000 UTC m=+71.874845514" watchObservedRunningTime="2026-09-09 20:53:53.251025964 +0000 UTC m=+71.875499906"
Sep 09 20:54:08.744366 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.744229    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7579dfff88-lj6xx"]
Sep 09 20:54:08.748398 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.748375    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.751529 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.751499    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 20:54:08.751529 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.751525    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 20:54:08.751801 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.751786    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2kzf5\""
Sep 09 20:54:08.751963 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.751804    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:54:08.752296 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.752279    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:54:08.753216 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.753200    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 20:54:08.753708 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.753688    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 20:54:08.771761 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.771735    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7579dfff88-lj6xx"]
Sep 09 20:54:08.824244 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.824218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.824244 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.824246    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcnx6\" (UniqueName: \"kubernetes.io/projected/20940e25-e1d8-41da-b973-55f00196c089-kube-api-access-fcnx6\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.824461 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.824272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-default-certificate\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.824461 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.824286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.824461 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.824334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-stats-auth\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.925291 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.925248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.925291 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.925287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcnx6\" (UniqueName: \"kubernetes.io/projected/20940e25-e1d8-41da-b973-55f00196c089-kube-api-access-fcnx6\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.925538 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.925321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-default-certificate\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.925538 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:08.925430    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:09.42540991 +0000 UTC m=+88.049883850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : configmap references non-existent config key: service-ca.crt
Sep 09 20:54:08.925538 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.925472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.925538 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.925517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-stats-auth\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.925785 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:08.925602    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:54:08.925785 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:08.925673    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:09.425655872 +0000 UTC m=+88.050129794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : secret "router-metrics-certs-default" not found
Sep 09 20:54:08.928055 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.928025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-default-certificate\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.928181 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.928025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-stats-auth\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:08.936477 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:08.936452    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcnx6\" (UniqueName: \"kubernetes.io/projected/20940e25-e1d8-41da-b973-55f00196c089-kube-api-access-fcnx6\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:09.429571 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:09.429531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:09.429571 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:09.429577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:09.429811 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:09.429695    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:54:09.429811 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:09.429709    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:10.429692133 +0000 UTC m=+89.054166052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : configmap references non-existent config key: service-ca.crt
Sep 09 20:54:09.429811 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:09.429737    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:10.429724163 +0000 UTC m=+89.054198104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : secret "router-metrics-certs-default" not found
Sep 09 20:54:10.434595 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:10.434547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:10.435056 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:10.434613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:10.435056 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:10.434726    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:12.434706784 +0000 UTC m=+91.059180720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : configmap references non-existent config key: service-ca.crt
Sep 09 20:54:10.435056 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:10.434740    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:54:10.435056 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:10.434794    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:12.43477956 +0000 UTC m=+91.059253502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : secret "router-metrics-certs-default" not found
Sep 09 20:54:12.450860 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.450798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:12.450860 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.450877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:12.451449 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:12.450960    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:16.450936408 +0000 UTC m=+95.075410331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : configmap references non-existent config key: service-ca.crt
Sep 09 20:54:12.451449 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:12.450974    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:54:12.451449 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:12.451016    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:16.451004569 +0000 UTC m=+95.075478487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : secret "router-metrics-certs-default" not found
Sep 09 20:54:12.551533 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.551503    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7hwww"]
Sep 09 20:54:12.554557 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.554541    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.557687 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.557660    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:54:12.557810 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.557660    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:54:12.557855 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.557806    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:54:12.559148 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.559128    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:54:12.559256 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.559130    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ksvhl\""
Sep 09 20:54:12.565759 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.565739    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7hwww"]
Sep 09 20:54:12.652550 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.652509    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.652727 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.652557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.652727 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.652581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/394b88bf-e239-4ffd-8c70-4c633a0292db-data-volume\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.652727 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.652639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmqgg\" (UniqueName: \"kubernetes.io/projected/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-api-access-fmqgg\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.652727 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.652689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/394b88bf-e239-4ffd-8c70-4c633a0292db-crio-socket\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.753093 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.752981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.753093 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.753047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.753349 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:12.753140    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:12.753349 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.753161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/394b88bf-e239-4ffd-8c70-4c633a0292db-data-volume\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.753349 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:12.753214    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:13.253193182 +0000 UTC m=+91.877667102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:12.753349 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:12.753150    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:12.753349 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.753244    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmqgg\" (UniqueName: \"kubernetes.io/projected/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-api-access-fmqgg\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.753349 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:12.753272    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:13.253260159 +0000 UTC m=+91.877734079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:12.753349 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.753310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/394b88bf-e239-4ffd-8c70-4c633a0292db-crio-socket\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.753655 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.753380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/394b88bf-e239-4ffd-8c70-4c633a0292db-crio-socket\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.753655 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.753500    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/394b88bf-e239-4ffd-8c70-4c633a0292db-data-volume\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:12.762122 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:12.762083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmqgg\" (UniqueName: \"kubernetes.io/projected/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-api-access-fmqgg\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:13.257094 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:13.257036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:13.257094 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:13.257099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:13.257391 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:13.257186    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:13.257391 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:13.257231    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:13.257391 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:13.257261    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:14.257242826 +0000 UTC m=+92.881716767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:13.257391 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:13.257280    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:14.257271343 +0000 UTC m=+92.881745261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:14.265802 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:14.265770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:14.266332 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:14.265815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:14.266332 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:14.265890    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:14.266332 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:14.265965    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:16.265947091 +0000 UTC m=+94.890421010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:14.266332 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:14.265995    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:14.266332 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:14.266032    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:16.266020571 +0000 UTC m=+94.890494513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:14.491418 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:14.491382    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mmcfd_b5c2c926-9449-4539-b439-06fb5d910856/dns-node-resolver/0.log"
Sep 09 20:54:15.493564 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:15.493533    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mxtw2_c468bb92-a81e-4d78-88c0-ae85f7c69561/node-ca/0.log"
Sep 09 20:54:15.890855 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:15.890827    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7plsw_393acb6a-39f1-46c0-9c3a-29f50af68eff/serve-healthcheck-canary/0.log"
Sep 09 20:54:16.277207 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:16.277097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:16.277207 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:16.277159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:16.277511 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:16.277230    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:16.277511 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:16.277285    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:16.277511 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:16.277311    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:20.277293168 +0000 UTC m=+98.901767112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:16.277511 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:16.277331    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:20.277320091 +0000 UTC m=+98.901794010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:16.478550 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:16.478488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:16.478748 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:16.478566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:16.478748 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:16.478648    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:24.478632318 +0000 UTC m=+103.103106237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : configmap references non-existent config key: service-ca.crt
Sep 09 20:54:16.478748 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:16.478708    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:54:16.478868 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:16.478763    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:24.478750893 +0000 UTC m=+103.103224812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : secret "router-metrics-certs-default" not found
Sep 09 20:54:20.102149 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:20.102084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:54:20.102541 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:20.102218    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:54:20.102541 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:20.102290    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls podName:17fc2904-8ec6-4d26-8f39-14abdbe1a1cc nodeName:}" failed. No retries permitted until 2026-09-09 20:55:24.102274404 +0000 UTC m=+162.726748322 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls") pod "dns-default-rjl4w" (UID: "17fc2904-8ec6-4d26-8f39-14abdbe1a1cc") : secret "dns-default-metrics-tls" not found
Sep 09 20:54:20.304611 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:20.304576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:20.304812 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:20.304624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:20.304812 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:20.304707    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:20.304812 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:20.304743    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:20.304812 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:20.304783    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:28.304766977 +0000 UTC m=+106.929240901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:20.304812 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:20.304798    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:28.304791229 +0000 UTC m=+106.929265147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:24.221240 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:24.221202    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q2w92"
Sep 09 20:54:24.536145 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:24.536026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:24.536145 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:24.536090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:24.536365 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:24.536234    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:40.536214849 +0000 UTC m=+119.160688767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : configmap references non-existent config key: service-ca.crt
Sep 09 20:54:24.536365 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:24.536254    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:54:24.536365 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:24.536348    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs podName:20940e25-e1d8-41da-b973-55f00196c089 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:40.536333506 +0000 UTC m=+119.160807425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs") pod "router-default-7579dfff88-lj6xx" (UID: "20940e25-e1d8-41da-b973-55f00196c089") : secret "router-metrics-certs-default" not found
Sep 09 20:54:25.169967 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.169923    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-lk7xf"]
Sep 09 20:54:25.172534 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.172517    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.175547 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.175525    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 20:54:25.176691 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.176661    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 20:54:25.176691 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.176680    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 20:54:25.176856 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.176703    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ds99p\""
Sep 09 20:54:25.176856 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.176748    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 20:54:25.187287 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.187264    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-lk7xf"]
Sep 09 20:54:25.240561 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.240515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdkc\" (UniqueName: \"kubernetes.io/projected/b1b1d962-4a1f-4c6e-a2ef-2a1703c14271-kube-api-access-qmdkc\") pod \"service-ca-75d99c484-lk7xf\" (UID: \"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271\") " pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.240561 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.240553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b1b1d962-4a1f-4c6e-a2ef-2a1703c14271-signing-cabundle\") pod \"service-ca-75d99c484-lk7xf\" (UID: \"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271\") " pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.240975 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.240597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b1b1d962-4a1f-4c6e-a2ef-2a1703c14271-signing-key\") pod \"service-ca-75d99c484-lk7xf\" (UID: \"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271\") " pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.341158 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.341084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b1b1d962-4a1f-4c6e-a2ef-2a1703c14271-signing-key\") pod \"service-ca-75d99c484-lk7xf\" (UID: \"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271\") " pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.341322 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.341208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdkc\" (UniqueName: \"kubernetes.io/projected/b1b1d962-4a1f-4c6e-a2ef-2a1703c14271-kube-api-access-qmdkc\") pod \"service-ca-75d99c484-lk7xf\" (UID: \"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271\") " pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.341322 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.341228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b1b1d962-4a1f-4c6e-a2ef-2a1703c14271-signing-cabundle\") pod \"service-ca-75d99c484-lk7xf\" (UID: \"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271\") " pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.341859 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.341799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b1b1d962-4a1f-4c6e-a2ef-2a1703c14271-signing-cabundle\") pod \"service-ca-75d99c484-lk7xf\" (UID: \"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271\") " pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.343664 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.343640    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b1b1d962-4a1f-4c6e-a2ef-2a1703c14271-signing-key\") pod \"service-ca-75d99c484-lk7xf\" (UID: \"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271\") " pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.358682 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.358652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdkc\" (UniqueName: \"kubernetes.io/projected/b1b1d962-4a1f-4c6e-a2ef-2a1703c14271-kube-api-access-qmdkc\") pod \"service-ca-75d99c484-lk7xf\" (UID: \"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271\") " pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.481633 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.481533    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-lk7xf"
Sep 09 20:54:25.762068 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:25.761985    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-lk7xf"]
Sep 09 20:54:25.765822 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:25.765795    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b1d962_4a1f_4c6e_a2ef_2a1703c14271.slice/crio-c68875be7cc931dba098011be50f3b47e6cf5fc1fa1976ddc4bec71de20721b6 WatchSource:0}: Error finding container c68875be7cc931dba098011be50f3b47e6cf5fc1fa1976ddc4bec71de20721b6: Status 404 returned error can't find the container with id c68875be7cc931dba098011be50f3b47e6cf5fc1fa1976ddc4bec71de20721b6
Sep 09 20:54:26.297547 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:26.297510    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-lk7xf" event={"ID":"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271","Type":"ContainerStarted","Data":"c68875be7cc931dba098011be50f3b47e6cf5fc1fa1976ddc4bec71de20721b6"}
Sep 09 20:54:28.303736 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:28.303697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-lk7xf" event={"ID":"b1b1d962-4a1f-4c6e-a2ef-2a1703c14271","Type":"ContainerStarted","Data":"9ad0290090df5f7e697cf8c2967404d2bb8632d96db34adb83c272131b0d7034"}
Sep 09 20:54:28.336596 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:28.336538    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-lk7xf" podStartSLOduration=1.408762405 podStartE2EDuration="3.336520628s" totalImagesPullingTime="1.927758223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:28.335947601 +0000 UTC m=+106.960421543" watchObservedRunningTime="2026-09-09 20:54:28.336520628 +0000 UTC m=+106.960994569"
Sep 09 20:54:28.370291 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:28.370260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:28.370438 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:28.370313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:28.370496 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:28.370477    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:28.370564 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:28.370553    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls podName:394b88bf-e239-4ffd-8c70-4c633a0292db nodeName:}" failed. No retries permitted until 2026-09-09 20:54:44.370533205 +0000 UTC m=+122.995007132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7hwww" (UID: "394b88bf-e239-4ffd-8c70-4c633a0292db") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:28.370855 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:28.370832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/394b88bf-e239-4ffd-8c70-4c633a0292db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:40.570375 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:40.570334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:40.570781 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:40.570451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:40.571143 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:40.571098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20940e25-e1d8-41da-b973-55f00196c089-service-ca-bundle\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:40.572925 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:40.572901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20940e25-e1d8-41da-b973-55f00196c089-metrics-certs\") pod \"router-default-7579dfff88-lj6xx\" (UID: \"20940e25-e1d8-41da-b973-55f00196c089\") " pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:40.857571 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:40.857475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:41.132958 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:41.132874    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7579dfff88-lj6xx"]
Sep 09 20:54:41.136187 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:41.136160    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20940e25_e1d8_41da_b973_55f00196c089.slice/crio-646289d7f0c263d4bd61731a5b5c5b59a9476a3987a95e02b29a75486d106229 WatchSource:0}: Error finding container 646289d7f0c263d4bd61731a5b5c5b59a9476a3987a95e02b29a75486d106229: Status 404 returned error can't find the container with id 646289d7f0c263d4bd61731a5b5c5b59a9476a3987a95e02b29a75486d106229
Sep 09 20:54:41.339656 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:41.339617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7579dfff88-lj6xx" event={"ID":"20940e25-e1d8-41da-b973-55f00196c089","Type":"ContainerStarted","Data":"43c89aa57620211a2e004005aec6c13ea2f6a08009c767efca43848598e4c2e8"}
Sep 09 20:54:41.339656 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:41.339664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7579dfff88-lj6xx" event={"ID":"20940e25-e1d8-41da-b973-55f00196c089","Type":"ContainerStarted","Data":"646289d7f0c263d4bd61731a5b5c5b59a9476a3987a95e02b29a75486d106229"}
Sep 09 20:54:41.363653 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:41.363591    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7579dfff88-lj6xx" podStartSLOduration=33.363568644 podStartE2EDuration="33.363568644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:41.362924043 +0000 UTC m=+119.987397985" watchObservedRunningTime="2026-09-09 20:54:41.363568644 +0000 UTC m=+119.988042592"
Sep 09 20:54:41.858031 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:41.857991    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:41.862629 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:41.862591    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:42.342189 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:42.342150    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:42.343311 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:42.343291    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7579dfff88-lj6xx"
Sep 09 20:54:44.399485 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:44.399449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:44.401973 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:44.401945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/394b88bf-e239-4ffd-8c70-4c633a0292db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7hwww\" (UID: \"394b88bf-e239-4ffd-8c70-4c633a0292db\") " pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:44.665918 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:44.665825    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ksvhl\""
Sep 09 20:54:44.674092 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:44.674066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7hwww"
Sep 09 20:54:44.951637 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:44.951565    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7hwww"]
Sep 09 20:54:44.954312 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:44.954282    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394b88bf_e239_4ffd_8c70_4c633a0292db.slice/crio-76a44b49b06801777230dc3c1c8ef302f84d902ba27eb7a92c62372f9bd9cef6 WatchSource:0}: Error finding container 76a44b49b06801777230dc3c1c8ef302f84d902ba27eb7a92c62372f9bd9cef6: Status 404 returned error can't find the container with id 76a44b49b06801777230dc3c1c8ef302f84d902ba27eb7a92c62372f9bd9cef6
Sep 09 20:54:45.350762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:45.350722    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7hwww" event={"ID":"394b88bf-e239-4ffd-8c70-4c633a0292db","Type":"ContainerStarted","Data":"7f70be4424781fbb70619607ff6291f55d13982e51af58bac5c8935e71a168f9"}
Sep 09 20:54:45.351017 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:45.350781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7hwww" event={"ID":"394b88bf-e239-4ffd-8c70-4c633a0292db","Type":"ContainerStarted","Data":"76a44b49b06801777230dc3c1c8ef302f84d902ba27eb7a92c62372f9bd9cef6"}
Sep 09 20:54:46.355993 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:46.355955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7hwww" event={"ID":"394b88bf-e239-4ffd-8c70-4c633a0292db","Type":"ContainerStarted","Data":"df7bf03e07c85a968562c09319495041afc6cc6e2d7778b72b90b26afffcc9c2"}
Sep 09 20:54:48.363938 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.363906    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7hwww" event={"ID":"394b88bf-e239-4ffd-8c70-4c633a0292db","Type":"ContainerStarted","Data":"10c293a6453962ac07a04b35bd08cf9b9b82b49830496bc2880351205cec25c3"}
Sep 09 20:54:48.385728 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.385682    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7hwww" podStartSLOduration=34.122055115 podStartE2EDuration="36.385668568s" totalImagesPullingTime="2.263613453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:54:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:48.384543137 +0000 UTC m=+127.009017078" watchObservedRunningTime="2026-09-09 20:54:48.385668568 +0000 UTC m=+127.010142509"
Sep 09 20:54:48.737683 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.737607    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55f6877b64-dvz5l"]
Sep 09 20:54:48.740873 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.740847    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-pkjg7"]
Sep 09 20:54:48.741011 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.740989    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.743756 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.743738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7"
Sep 09 20:54:48.747474 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.747456    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:54:48.747805 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.747782    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 20:54:48.748212 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.748191    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 20:54:48.750688 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.750668    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vrdjz\""
Sep 09 20:54:48.750796 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.750777    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:54:48.751166 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.751152    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xb6jr\""
Sep 09 20:54:48.754740 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.754723    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:54:48.756731 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.756712    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:54:48.769365 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.769346    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-pkjg7"]
Sep 09 20:54:48.772092 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.772069    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55f6877b64-dvz5l"]
Sep 09 20:54:48.794524 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.794494    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"]
Sep 09 20:54:48.797491 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.797475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"
Sep 09 20:54:48.808159 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.808133    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:54:48.808317 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.808299    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 20:54:48.808392 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.808377    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:54:48.808451 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.808406    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:54:48.808509 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.808454    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-fbrnr\""
Sep 09 20:54:48.831527 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-installation-pull-secrets\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.831527 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb34e84e-d11f-4ea2-97f2-e79ada097ce7-nginx-conf\") pod \"networking-console-plugin-79977f9844-pkjg7\" (UID: \"eb34e84e-d11f-4ea2-97f2-e79ada097ce7\") " pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7"
Sep 09 20:54:48.831764 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-certificates\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.831764 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831572    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4q52\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-kube-api-access-q4q52\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.831764 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-tls\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.831764 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb34e84e-d11f-4ea2-97f2-e79ada097ce7-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-pkjg7\" (UID: \"eb34e84e-d11f-4ea2-97f2-e79ada097ce7\") " pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7"
Sep 09 20:54:48.831918 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-image-registry-private-configuration\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.831918 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831801    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88296d1b-b71d-43fc-81aa-33afac2044ce-ca-trust-extracted\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.831918 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-bound-sa-token\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.831918 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.831861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-trusted-ca\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.876835 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.876786    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"]
Sep 09 20:54:48.932918 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.932875    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-tls\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.933135 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.932946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb34e84e-d11f-4ea2-97f2-e79ada097ce7-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-pkjg7\" (UID: \"eb34e84e-d11f-4ea2-97f2-e79ada097ce7\") " pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7"
Sep 09 20:54:48.933135 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzqm\" (UniqueName: \"kubernetes.io/projected/c2aa3741-3343-4785-9e85-7eefbb56a472-kube-api-access-8hzqm\") pod \"managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v\" (UID: \"c2aa3741-3343-4785-9e85-7eefbb56a472\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"
Sep 09 20:54:48.933261 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-image-registry-private-configuration\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.933261 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88296d1b-b71d-43fc-81aa-33afac2044ce-ca-trust-extracted\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.933261 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-bound-sa-token\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.933261 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-trusted-ca\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.933455 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-installation-pull-secrets\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.933455 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2aa3741-3343-4785-9e85-7eefbb56a472-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v\" (UID: \"c2aa3741-3343-4785-9e85-7eefbb56a472\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"
Sep 09 20:54:48.933455 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb34e84e-d11f-4ea2-97f2-e79ada097ce7-nginx-conf\") pod \"networking-console-plugin-79977f9844-pkjg7\" (UID: \"eb34e84e-d11f-4ea2-97f2-e79ada097ce7\") " pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7"
Sep 09 20:54:48.933590 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-certificates\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.933590 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4q52\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-kube-api-access-q4q52\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.933590 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.933566    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88296d1b-b71d-43fc-81aa-33afac2044ce-ca-trust-extracted\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.934578 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.934309    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb34e84e-d11f-4ea2-97f2-e79ada097ce7-nginx-conf\") pod \"networking-console-plugin-79977f9844-pkjg7\" (UID: \"eb34e84e-d11f-4ea2-97f2-e79ada097ce7\") " pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7"
Sep 09 20:54:48.934811 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.934630    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-certificates\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.935545 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.935511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-trusted-ca\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.935993 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.935971    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55f6877b64-dvz5l"]
Sep 09 20:54:48.936138 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.935975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb34e84e-d11f-4ea2-97f2-e79ada097ce7-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-pkjg7\" (UID: \"eb34e84e-d11f-4ea2-97f2-e79ada097ce7\") " pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7"
Sep 09 20:54:48.936138 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.936098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-image-registry-private-configuration\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.936261 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.936196    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-tls\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.936311 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:48.936292    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token image-registry-private-configuration installation-pull-secrets kube-api-access-q4q52 registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-55f6877b64-dvz5l" podUID="88296d1b-b71d-43fc-81aa-33afac2044ce"
Sep 09 20:54:48.936561 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.936541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-installation-pull-secrets\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.966294 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.966261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4q52\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-kube-api-access-q4q52\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:48.966420 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:48.966395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-bound-sa-token\") pod \"image-registry-55f6877b64-dvz5l\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") " pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:49.034944 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.034857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzqm\" (UniqueName: \"kubernetes.io/projected/c2aa3741-3343-4785-9e85-7eefbb56a472-kube-api-access-8hzqm\") pod \"managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v\" (UID: \"c2aa3741-3343-4785-9e85-7eefbb56a472\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"
Sep 09 20:54:49.034944 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.034895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2aa3741-3343-4785-9e85-7eefbb56a472-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v\" (UID: \"c2aa3741-3343-4785-9e85-7eefbb56a472\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"
Sep 09 20:54:49.037449 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.037430    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2aa3741-3343-4785-9e85-7eefbb56a472-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v\" (UID: \"c2aa3741-3343-4785-9e85-7eefbb56a472\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"
Sep 09 20:54:49.056602 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.056575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzqm\" (UniqueName: \"kubernetes.io/projected/c2aa3741-3343-4785-9e85-7eefbb56a472-kube-api-access-8hzqm\") pod \"managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v\" (UID: \"c2aa3741-3343-4785-9e85-7eefbb56a472\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"
Sep 09 20:54:49.058412 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.058393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7"
Sep 09 20:54:49.122479 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.122452    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"
Sep 09 20:54:49.344340 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.344306    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-pkjg7"]
Sep 09 20:54:49.348486 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:49.348461    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb34e84e_d11f_4ea2_97f2_e79ada097ce7.slice/crio-999005c4dc79efc4d528e7f40a28b9dadbb668fe51a3904948d2d9641edc8688 WatchSource:0}: Error finding container 999005c4dc79efc4d528e7f40a28b9dadbb668fe51a3904948d2d9641edc8688: Status 404 returned error can't find the container with id 999005c4dc79efc4d528e7f40a28b9dadbb668fe51a3904948d2d9641edc8688
Sep 09 20:54:49.368122 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.368072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7" event={"ID":"eb34e84e-d11f-4ea2-97f2-e79ada097ce7","Type":"ContainerStarted","Data":"999005c4dc79efc4d528e7f40a28b9dadbb668fe51a3904948d2d9641edc8688"}
Sep 09 20:54:49.368122 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.368090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:49.375791 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.375770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:49.420177 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.420148    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v"]
Sep 09 20:54:49.422719 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:49.422695    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2aa3741_3343_4785_9e85_7eefbb56a472.slice/crio-16b03997ca7066746673e7824a2cd906e34497baf4c144ca6292fd59cbec5ccd WatchSource:0}: Error finding container 16b03997ca7066746673e7824a2cd906e34497baf4c144ca6292fd59cbec5ccd: Status 404 returned error can't find the container with id 16b03997ca7066746673e7824a2cd906e34497baf4c144ca6292fd59cbec5ccd
Sep 09 20:54:49.538782 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.538738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-installation-pull-secrets\") pod \"88296d1b-b71d-43fc-81aa-33afac2044ce\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") "
Sep 09 20:54:49.538782 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.538786    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-tls\") pod \"88296d1b-b71d-43fc-81aa-33afac2044ce\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") "
Sep 09 20:54:49.539017 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.538809    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-trusted-ca\") pod \"88296d1b-b71d-43fc-81aa-33afac2044ce\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") "
Sep 09 20:54:49.539017 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.538832    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-bound-sa-token\") pod \"88296d1b-b71d-43fc-81aa-33afac2044ce\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") "
Sep 09 20:54:49.539017 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.538889    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4q52\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-kube-api-access-q4q52\") pod \"88296d1b-b71d-43fc-81aa-33afac2044ce\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") "
Sep 09 20:54:49.539017 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.538925    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-image-registry-private-configuration\") pod \"88296d1b-b71d-43fc-81aa-33afac2044ce\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") "
Sep 09 20:54:49.539017 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.538958    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88296d1b-b71d-43fc-81aa-33afac2044ce-ca-trust-extracted\") pod \"88296d1b-b71d-43fc-81aa-33afac2044ce\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") "
Sep 09 20:54:49.539017 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.538985    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-certificates\") pod \"88296d1b-b71d-43fc-81aa-33afac2044ce\" (UID: \"88296d1b-b71d-43fc-81aa-33afac2044ce\") "
Sep 09 20:54:49.539422 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.539394    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "88296d1b-b71d-43fc-81aa-33afac2044ce" (UID: "88296d1b-b71d-43fc-81aa-33afac2044ce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:49.539645 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.539525    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "88296d1b-b71d-43fc-81aa-33afac2044ce" (UID: "88296d1b-b71d-43fc-81aa-33afac2044ce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:49.539645 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.539593    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88296d1b-b71d-43fc-81aa-33afac2044ce-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "88296d1b-b71d-43fc-81aa-33afac2044ce" (UID: "88296d1b-b71d-43fc-81aa-33afac2044ce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:49.541564 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.541537    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-kube-api-access-q4q52" (OuterVolumeSpecName: "kube-api-access-q4q52") pod "88296d1b-b71d-43fc-81aa-33afac2044ce" (UID: "88296d1b-b71d-43fc-81aa-33afac2044ce"). InnerVolumeSpecName "kube-api-access-q4q52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:49.541674 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.541571    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "88296d1b-b71d-43fc-81aa-33afac2044ce" (UID: "88296d1b-b71d-43fc-81aa-33afac2044ce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:49.541674 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.541604    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "88296d1b-b71d-43fc-81aa-33afac2044ce" (UID: "88296d1b-b71d-43fc-81aa-33afac2044ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:49.541875 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.541853    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "88296d1b-b71d-43fc-81aa-33afac2044ce" (UID: "88296d1b-b71d-43fc-81aa-33afac2044ce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:49.541949 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.541871    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "88296d1b-b71d-43fc-81aa-33afac2044ce" (UID: "88296d1b-b71d-43fc-81aa-33afac2044ce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:49.639736 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.639653    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-image-registry-private-configuration\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:54:49.639736 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.639681    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88296d1b-b71d-43fc-81aa-33afac2044ce-ca-trust-extracted\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:54:49.639736 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.639692    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-certificates\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:54:49.639736 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.639700    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88296d1b-b71d-43fc-81aa-33afac2044ce-installation-pull-secrets\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:54:49.639736 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.639709    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-registry-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:54:49.639736 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.639716    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88296d1b-b71d-43fc-81aa-33afac2044ce-trusted-ca\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:54:49.639736 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.639724    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-bound-sa-token\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:54:49.639736 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:49.639731    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4q52\" (UniqueName: \"kubernetes.io/projected/88296d1b-b71d-43fc-81aa-33afac2044ce-kube-api-access-q4q52\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:54:50.374686 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:50.374651    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55f6877b64-dvz5l"
Sep 09 20:54:50.375152 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:50.374652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v" event={"ID":"c2aa3741-3343-4785-9e85-7eefbb56a472","Type":"ContainerStarted","Data":"16b03997ca7066746673e7824a2cd906e34497baf4c144ca6292fd59cbec5ccd"}
Sep 09 20:54:50.412896 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:50.412821    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55f6877b64-dvz5l"]
Sep 09 20:54:50.414566 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:50.414540    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55f6877b64-dvz5l"]
Sep 09 20:54:51.379230 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:51.379181    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7" event={"ID":"eb34e84e-d11f-4ea2-97f2-e79ada097ce7","Type":"ContainerStarted","Data":"e0e63aeb1a43f87857928c30931c51fa6188e7dd8e742d3303e90ff0ab32f4ed"}
Sep 09 20:54:51.425679 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:51.425617    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-pkjg7" podStartSLOduration=2.084446902 podStartE2EDuration="3.425598363s" totalImagesPullingTime="1.341151461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:51.41740036 +0000 UTC m=+130.041874303" watchObservedRunningTime="2026-09-09 20:54:51.425598363 +0000 UTC m=+130.050072306"
Sep 09 20:54:51.660049 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:51.659961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:54:51.662657 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:51.662622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43fba51-6814-4c9e-a5c8-a1dd796b7386-metrics-certs\") pod \"network-metrics-daemon-29hf4\" (UID: \"d43fba51-6814-4c9e-a5c8-a1dd796b7386\") " pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:54:51.681291 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:51.680818    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2slq\""
Sep 09 20:54:51.686740 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:51.686694    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29hf4"
Sep 09 20:54:51.864123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:51.864064    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88296d1b-b71d-43fc-81aa-33afac2044ce" path="/var/lib/kubelet/pods/88296d1b-b71d-43fc-81aa-33afac2044ce/volumes"
Sep 09 20:54:52.199969 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.199819    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29hf4"]
Sep 09 20:54:52.202332 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:52.202308    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43fba51_6814_4c9e_a5c8_a1dd796b7386.slice/crio-bd177236ed8dfbd30ed77a69601aa6cc34a5e6340e74530b9980d0bc24a31138 WatchSource:0}: Error finding container bd177236ed8dfbd30ed77a69601aa6cc34a5e6340e74530b9980d0bc24a31138: Status 404 returned error can't find the container with id bd177236ed8dfbd30ed77a69601aa6cc34a5e6340e74530b9980d0bc24a31138
Sep 09 20:54:52.384512 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.384472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v" event={"ID":"c2aa3741-3343-4785-9e85-7eefbb56a472","Type":"ContainerStarted","Data":"f316b1231de66d19e3eb05b88fe7deab4ea413d94cd73ac0c10f4908c6f919e3"}
Sep 09 20:54:52.385741 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.385714    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29hf4" event={"ID":"d43fba51-6814-4c9e-a5c8-a1dd796b7386","Type":"ContainerStarted","Data":"bd177236ed8dfbd30ed77a69601aa6cc34a5e6340e74530b9980d0bc24a31138"}
Sep 09 20:54:52.409458 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.409407    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74ff9fd9d8-swh6v" podStartSLOduration=1.839861593 podStartE2EDuration="4.409386756s" totalImagesPullingTime="2.569525163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:52.40853252 +0000 UTC m=+131.033006460" watchObservedRunningTime="2026-09-09 20:54:52.409386756 +0000 UTC m=+131.033860698"
Sep 09 20:54:52.443882 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.443847    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"]
Sep 09 20:54:52.447218 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.447200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.450797 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.450715    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:54:52.450797 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.450757    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 20:54:52.450940 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.450867    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:54:52.451043 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.451004    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:54:52.451136 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.451056    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 20:54:52.451136 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.451066    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nm9d8\""
Sep 09 20:54:52.458797 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.458775    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"]
Sep 09 20:54:52.565964 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.565920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b35118e-dda0-43f2-bf20-312ebbd27efa-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.565964 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.565972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b35118e-dda0-43f2-bf20-312ebbd27efa-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.566240 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.566012    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7bb9\" (UniqueName: \"kubernetes.io/projected/8b35118e-dda0-43f2-bf20-312ebbd27efa-kube-api-access-t7bb9\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.566240 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.566162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b35118e-dda0-43f2-bf20-312ebbd27efa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.666951 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.666907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b35118e-dda0-43f2-bf20-312ebbd27efa-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.667167 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.666957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b35118e-dda0-43f2-bf20-312ebbd27efa-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.667167 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.666993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7bb9\" (UniqueName: \"kubernetes.io/projected/8b35118e-dda0-43f2-bf20-312ebbd27efa-kube-api-access-t7bb9\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.667167 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.667055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b35118e-dda0-43f2-bf20-312ebbd27efa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.667762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.667739    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b35118e-dda0-43f2-bf20-312ebbd27efa-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.669961 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.669934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b35118e-dda0-43f2-bf20-312ebbd27efa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.670155 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.670133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b35118e-dda0-43f2-bf20-312ebbd27efa-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.679406 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.679376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7bb9\" (UniqueName: \"kubernetes.io/projected/8b35118e-dda0-43f2-bf20-312ebbd27efa-kube-api-access-t7bb9\") pod \"prometheus-operator-6f4946db56-wlbb2\" (UID: \"8b35118e-dda0-43f2-bf20-312ebbd27efa\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:52.757520 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:52.757440    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"
Sep 09 20:54:53.038757 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:53.038604    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wlbb2"]
Sep 09 20:54:53.058332 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:53.058304    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b35118e_dda0_43f2_bf20_312ebbd27efa.slice/crio-35a27cad81fc7e20a3dfe25fd879c72214205774dba35c2f7c9b9eace40f61dd WatchSource:0}: Error finding container 35a27cad81fc7e20a3dfe25fd879c72214205774dba35c2f7c9b9eace40f61dd: Status 404 returned error can't find the container with id 35a27cad81fc7e20a3dfe25fd879c72214205774dba35c2f7c9b9eace40f61dd
Sep 09 20:54:53.391607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:53.391570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29hf4" event={"ID":"d43fba51-6814-4c9e-a5c8-a1dd796b7386","Type":"ContainerStarted","Data":"bfd319369ca44e5b7eceb2e6c22047593164cf6d1aa55f40d8e33e37bc4f5424"}
Sep 09 20:54:53.392032 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:53.391623    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29hf4" event={"ID":"d43fba51-6814-4c9e-a5c8-a1dd796b7386","Type":"ContainerStarted","Data":"ef822a8d8d3ce82682dd9e51ffdbfd4f811f022fe2507df6cafe3195dcff5acb"}
Sep 09 20:54:53.392718 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:53.392688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2" event={"ID":"8b35118e-dda0-43f2-bf20-312ebbd27efa","Type":"ContainerStarted","Data":"35a27cad81fc7e20a3dfe25fd879c72214205774dba35c2f7c9b9eace40f61dd"}
Sep 09 20:54:53.411306 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:53.411253    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-29hf4" podStartSLOduration=130.513620046 podStartE2EDuration="2m11.411238334s" totalImagesPullingTime="897.618288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:53.410309249 +0000 UTC m=+132.034783191" watchObservedRunningTime="2026-09-09 20:54:53.411238334 +0000 UTC m=+132.035712274"
Sep 09 20:54:55.404286 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:55.404246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2" event={"ID":"8b35118e-dda0-43f2-bf20-312ebbd27efa","Type":"ContainerStarted","Data":"e8d12aaf78772afb376fa033ba87e69465c51d1e92b73fb4b112d0405675b874"}
Sep 09 20:54:55.404657 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:55.404316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2" event={"ID":"8b35118e-dda0-43f2-bf20-312ebbd27efa","Type":"ContainerStarted","Data":"7123cae630d86ef92bce98866ead35652410a685ea71a82fdb7666b74f4cbf07"}
Sep 09 20:54:55.424702 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:55.424651    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-wlbb2" podStartSLOduration=2.120369406 podStartE2EDuration="3.424635125s" totalImagesPullingTime="1.304265719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:55.42376 +0000 UTC m=+134.048233953" watchObservedRunningTime="2026-09-09 20:54:55.424635125 +0000 UTC m=+134.049109066"
Sep 09 20:54:57.761000 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.760962    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"]
Sep 09 20:54:57.764692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.764668    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:57.767657 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.767621    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4s5gt\""
Sep 09 20:54:57.770256 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.770224    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 20:54:57.770582 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.770558    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:54:57.770699 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.770681    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c9c6k"]
Sep 09 20:54:57.774087 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.773894    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:57.775078 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.775050    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"]
Sep 09 20:54:57.776451 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.776428    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:54:57.776544 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.776527    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tqnv2\""
Sep 09 20:54:57.777155 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.777135    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:54:57.777268 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.777191    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:54:57.778495 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.778476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:57.781210 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.781187    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 20:54:57.781512 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.781487    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:54:57.781594 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.781572    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"]
Sep 09 20:54:57.781702 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.781682    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v8km2\""
Sep 09 20:54:57.781702 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.781690    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 20:54:57.794915 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.794894    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"]
Sep 09 20:54:57.906786 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.906737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:57.906786 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.906787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/025696b8-f9ab-44ea-a08a-060bc44b12ec-root\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:57.907003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.906806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:57.907003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.906837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebea8244-c7f2-483d-83ab-3db65b3b770a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:57.907003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.906900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025696b8-f9ab-44ea-a08a-060bc44b12ec-metrics-client-ca\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:57.907003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.906973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:57.907183 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:57.907183 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:57.907183 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/025696b8-f9ab-44ea-a08a-060bc44b12ec-sys\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:57.907183 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:57.907183 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ebea8244-c7f2-483d-83ab-3db65b3b770a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:57.907411 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:57.907411 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q9fp\" (UniqueName: \"kubernetes.io/projected/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-api-access-4q9fp\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:57.907411 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5gc\" (UniqueName: \"kubernetes.io/projected/025696b8-f9ab-44ea-a08a-060bc44b12ec-kube-api-access-7f5gc\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:57.907411 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:57.907411 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-wtmp\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:57.907411 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k6qh\" (UniqueName: \"kubernetes.io/projected/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-kube-api-access-4k6qh\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:57.907654 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-textfile\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:57.907654 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:57.907441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-tls\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.008759 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.008759 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/025696b8-f9ab-44ea-a08a-060bc44b12ec-sys\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ebea8244-c7f2-483d-83ab-3db65b3b770a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9fp\" (UniqueName: \"kubernetes.io/projected/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-api-access-4q9fp\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:58.008902    2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5gc\" (UniqueName: \"kubernetes.io/projected/025696b8-f9ab-44ea-a08a-060bc44b12ec-kube-api-access-7f5gc\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:58.008979    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-tls podName:ebea8244-c7f2-483d-83ab-3db65b3b770a nodeName:}" failed. No retries permitted until 2026-09-09 20:54:58.50895785 +0000 UTC m=+137.133431777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-hkqwt" (UID: "ebea8244-c7f2-483d-83ab-3db65b3b770a") : secret "kube-state-metrics-tls" not found
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:58.009001    2583 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 20:54:58.009016 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-wtmp\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6qh\" (UniqueName: \"kubernetes.io/projected/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-kube-api-access-4k6qh\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:54:58.009058    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-openshift-state-metrics-tls podName:f8dfea5c-68bf-4e9f-b320-269f8cf7775b nodeName:}" failed. No retries permitted until 2026-09-09 20:54:58.509042021 +0000 UTC m=+137.133515939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-h7zvw" (UID: "f8dfea5c-68bf-4e9f-b320-269f8cf7775b") : secret "openshift-state-metrics-tls" not found
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-textfile\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-tls\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/025696b8-f9ab-44ea-a08a-060bc44b12ec-root\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebea8244-c7f2-483d-83ab-3db65b3b770a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025696b8-f9ab-44ea-a08a-060bc44b12ec-metrics-client-ca\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.009607 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-wtmp\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.010153 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/025696b8-f9ab-44ea-a08a-060bc44b12ec-root\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.010153 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009656    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.010153 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.008901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/025696b8-f9ab-44ea-a08a-060bc44b12ec-sys\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.010153 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.010153 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.009913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-textfile\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.010153 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.010048    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.010153 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.010052    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ebea8244-c7f2-483d-83ab-3db65b3b770a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.010548 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.010523    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebea8244-c7f2-483d-83ab-3db65b3b770a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.010769 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.010705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025696b8-f9ab-44ea-a08a-060bc44b12ec-metrics-client-ca\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.012429 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.012376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-tls\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.012647 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.012629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.013013 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.012981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025696b8-f9ab-44ea-a08a-060bc44b12ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.013013 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.012989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.024200 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.024175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5gc\" (UniqueName: \"kubernetes.io/projected/025696b8-f9ab-44ea-a08a-060bc44b12ec-kube-api-access-7f5gc\") pod \"node-exporter-c9c6k\" (UID: \"025696b8-f9ab-44ea-a08a-060bc44b12ec\") " pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.024570 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.024551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9fp\" (UniqueName: \"kubernetes.io/projected/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-api-access-4q9fp\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.026542 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.026522    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6qh\" (UniqueName: \"kubernetes.io/projected/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-kube-api-access-4k6qh\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.088853 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.088817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9c6k"
Sep 09 20:54:58.100180 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:58.100152    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025696b8_f9ab_44ea_a08a_060bc44b12ec.slice/crio-9526db2316ae5d5d119ed7c8d11aa85ff50e4e884653e5716534243baf98c20f WatchSource:0}: Error finding container 9526db2316ae5d5d119ed7c8d11aa85ff50e4e884653e5716534243baf98c20f: Status 404 returned error can't find the container with id 9526db2316ae5d5d119ed7c8d11aa85ff50e4e884653e5716534243baf98c20f
Sep 09 20:54:58.415003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.414960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9c6k" event={"ID":"025696b8-f9ab-44ea-a08a-060bc44b12ec","Type":"ContainerStarted","Data":"9526db2316ae5d5d119ed7c8d11aa85ff50e4e884653e5716534243baf98c20f"}
Sep 09 20:54:58.512767 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.512725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.512767 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.512773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.515481 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.515432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8dfea5c-68bf-4e9f-b320-269f8cf7775b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-h7zvw\" (UID: \"f8dfea5c-68bf-4e9f-b320-269f8cf7775b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.515481 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.515478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea8244-c7f2-483d-83ab-3db65b3b770a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hkqwt\" (UID: \"ebea8244-c7f2-483d-83ab-3db65b3b770a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.676043 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.675948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"
Sep 09 20:54:58.695247 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.695217    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"
Sep 09 20:54:58.972963 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.972763    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw"]
Sep 09 20:54:58.975784 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:58.975756    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dfea5c_68bf_4e9f_b320_269f8cf7775b.slice/crio-e5634f3e40cb10021345a8162055d2ccc4717e60f27faba2517f34ef9416391c WatchSource:0}: Error finding container e5634f3e40cb10021345a8162055d2ccc4717e60f27faba2517f34ef9416391c: Status 404 returned error can't find the container with id e5634f3e40cb10021345a8162055d2ccc4717e60f27faba2517f34ef9416391c
Sep 09 20:54:58.996164 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:58.996018    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt"]
Sep 09 20:54:58.999341 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:54:58.999311    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebea8244_c7f2_483d_83ab_3db65b3b770a.slice/crio-6292af74ac17412231ff8c966107f270fe8ae9af48988af270c3fbf5d2650c61 WatchSource:0}: Error finding container 6292af74ac17412231ff8c966107f270fe8ae9af48988af270c3fbf5d2650c61: Status 404 returned error can't find the container with id 6292af74ac17412231ff8c966107f270fe8ae9af48988af270c3fbf5d2650c61
Sep 09 20:54:59.419030 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:59.418999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt" event={"ID":"ebea8244-c7f2-483d-83ab-3db65b3b770a","Type":"ContainerStarted","Data":"6292af74ac17412231ff8c966107f270fe8ae9af48988af270c3fbf5d2650c61"}
Sep 09 20:54:59.421198 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:59.421176    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw" event={"ID":"f8dfea5c-68bf-4e9f-b320-269f8cf7775b","Type":"ContainerStarted","Data":"ba733d1078bc8cc82ad2a52adccb364cd50081c72fe856bd825c67b65534ca34"}
Sep 09 20:54:59.421352 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:59.421230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw" event={"ID":"f8dfea5c-68bf-4e9f-b320-269f8cf7775b","Type":"ContainerStarted","Data":"40630148022eb9dc2086e977eee1e3cc91dc3cd5fd2639e5df4f321418013f2d"}
Sep 09 20:54:59.421352 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:54:59.421263    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw" event={"ID":"f8dfea5c-68bf-4e9f-b320-269f8cf7775b","Type":"ContainerStarted","Data":"e5634f3e40cb10021345a8162055d2ccc4717e60f27faba2517f34ef9416391c"}
Sep 09 20:55:00.426814 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.426782    2583 generic.go:356] "Generic (PLEG): container finished" podID="025696b8-f9ab-44ea-a08a-060bc44b12ec" containerID="bc5a3e92ebbc2717dd9aa1c2e4a019b859b2e0a829fdcdec62d2aee323395f02" exitCode=0
Sep 09 20:55:00.427235 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.426873    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9c6k" event={"ID":"025696b8-f9ab-44ea-a08a-060bc44b12ec","Type":"ContainerDied","Data":"bc5a3e92ebbc2717dd9aa1c2e4a019b859b2e0a829fdcdec62d2aee323395f02"}
Sep 09 20:55:00.775654 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.775621    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"]
Sep 09 20:55:00.779241 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.779213    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.782275 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.782238    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 20:55:00.782414 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.782394    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 20:55:00.782551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.782531    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 20:55:00.782777 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.782760    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jl2kl\""
Sep 09 20:55:00.782844 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.782774    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 20:55:00.783069 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.783044    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 20:55:00.783462 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.783445    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9hqsamq379acv\""
Sep 09 20:55:00.794698 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.794608    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"]
Sep 09 20:55:00.830501 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.830461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-tls\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.830501 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.830504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.830755 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.830530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-grpc-tls\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.830755 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.830546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6508432d-5e5c-4400-83a0-dad1cfe1c39f-metrics-client-ca\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.830755 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.830563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.830755 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.830590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.830755 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.830614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xvvs\" (UniqueName: \"kubernetes.io/projected/6508432d-5e5c-4400-83a0-dad1cfe1c39f-kube-api-access-9xvvs\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.830755 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.830632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.931265 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.931228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.931458 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.931276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-grpc-tls\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.931458 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.931297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6508432d-5e5c-4400-83a0-dad1cfe1c39f-metrics-client-ca\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.931458 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.931440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.931627 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.931504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.931627 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.931550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xvvs\" (UniqueName: \"kubernetes.io/projected/6508432d-5e5c-4400-83a0-dad1cfe1c39f-kube-api-access-9xvvs\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.931727 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.931626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.931727 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.931688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-tls\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.932640 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.932002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6508432d-5e5c-4400-83a0-dad1cfe1c39f-metrics-client-ca\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.935001 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.934972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.935098 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.934974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.935432 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.935405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.935507 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.935461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-tls\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.935563 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.935520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-grpc-tls\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.937015 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.936964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6508432d-5e5c-4400-83a0-dad1cfe1c39f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:00.941228 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:00.941208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xvvs\" (UniqueName: \"kubernetes.io/projected/6508432d-5e5c-4400-83a0-dad1cfe1c39f-kube-api-access-9xvvs\") pod \"thanos-querier-694bd65f4d-gbcwm\" (UID: \"6508432d-5e5c-4400-83a0-dad1cfe1c39f\") " pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:01.091532 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.091470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:01.372929 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.372889    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"]
Sep 09 20:55:01.375908 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:55:01.375883    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6508432d_5e5c_4400_83a0_dad1cfe1c39f.slice/crio-727895b211e957ddeeee203147299a627b7bab73a0301404553075b66fd7b8ad WatchSource:0}: Error finding container 727895b211e957ddeeee203147299a627b7bab73a0301404553075b66fd7b8ad: Status 404 returned error can't find the container with id 727895b211e957ddeeee203147299a627b7bab73a0301404553075b66fd7b8ad
Sep 09 20:55:01.432679 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.432643    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw" event={"ID":"f8dfea5c-68bf-4e9f-b320-269f8cf7775b","Type":"ContainerStarted","Data":"795eccd1e91a13d5f1239dc43dd5e50a34c664ea45f028899810fe5825e34c4f"}
Sep 09 20:55:01.433969 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.433949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm" event={"ID":"6508432d-5e5c-4400-83a0-dad1cfe1c39f","Type":"ContainerStarted","Data":"727895b211e957ddeeee203147299a627b7bab73a0301404553075b66fd7b8ad"}
Sep 09 20:55:01.436408 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.436381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9c6k" event={"ID":"025696b8-f9ab-44ea-a08a-060bc44b12ec","Type":"ContainerStarted","Data":"89de4dd9db7d9ef7d782fe7bd0f1dceb8b39ec88c2b4dc5c4adf1d73c552a1b9"}
Sep 09 20:55:01.436519 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.436432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9c6k" event={"ID":"025696b8-f9ab-44ea-a08a-060bc44b12ec","Type":"ContainerStarted","Data":"d28a067c3667173c16756e79aba97c9772c34292a44b0a8f0640c2678aca8542"}
Sep 09 20:55:01.438523 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.438501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt" event={"ID":"ebea8244-c7f2-483d-83ab-3db65b3b770a","Type":"ContainerStarted","Data":"986a8d054d0657b0d1a7f2b06336e11017a75f2d2a7b6beb779096d7fc3e04af"}
Sep 09 20:55:01.438643 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.438549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt" event={"ID":"ebea8244-c7f2-483d-83ab-3db65b3b770a","Type":"ContainerStarted","Data":"bc7f6825f9dc0f9f811878b013dc0110093cb169bccc1122cb71d70a8ecbfb36"}
Sep 09 20:55:01.438643 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.438577    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt" event={"ID":"ebea8244-c7f2-483d-83ab-3db65b3b770a","Type":"ContainerStarted","Data":"e9940a92390c9cf677b054fe75d41700ef853d0ec60d54b355d2d5f37de63064"}
Sep 09 20:55:01.451930 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.451872    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-h7zvw" podStartSLOduration=3.096468332 podStartE2EDuration="4.451855531s" totalImagesPullingTime="1.355387199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:01.450907801 +0000 UTC m=+140.075381741" watchObservedRunningTime="2026-09-09 20:55:01.451855531 +0000 UTC m=+140.076329476"
Sep 09 20:55:01.475629 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.475575    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c9c6k" podStartSLOduration=3.109023341 podStartE2EDuration="4.475551605s" totalImagesPullingTime="1.295813727s" totalInitContainerRuntime="70.714537ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:01.474905348 +0000 UTC m=+140.099379289" watchObservedRunningTime="2026-09-09 20:55:01.475551605 +0000 UTC m=+140.100025580"
Sep 09 20:55:01.495123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:01.495061    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hkqwt" podStartSLOduration=2.992859428 podStartE2EDuration="4.49504542s" totalImagesPullingTime="1.502185992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:01.49371471 +0000 UTC m=+140.118188652" watchObservedRunningTime="2026-09-09 20:55:01.49504542 +0000 UTC m=+140.119519361"
Sep 09 20:55:02.267167 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.267132    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6cc84b76b8-2nclq"]
Sep 09 20:55:02.276124 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.276066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.279435 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.279402    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 20:55:02.280725 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.280676    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 20:55:02.280874 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.280799    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 20:55:02.280874 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.280828    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 20:55:02.280997 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.280981    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jhxgb\""
Sep 09 20:55:02.281077 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.281051    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d0kafg9f5g91a\""
Sep 09 20:55:02.283229 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.283202    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cc84b76b8-2nclq"]
Sep 09 20:55:02.342865 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.342824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/de8df011-c523-45a1-8266-dc8cfe411749-metrics-server-audit-profiles\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.343038 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.342953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/de8df011-c523-45a1-8266-dc8cfe411749-secret-metrics-server-tls\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.343145 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.343036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9c8\" (UniqueName: \"kubernetes.io/projected/de8df011-c523-45a1-8266-dc8cfe411749-kube-api-access-4x9c8\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.343145 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.343114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/de8df011-c523-45a1-8266-dc8cfe411749-audit-log\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.343256 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.343183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/de8df011-c523-45a1-8266-dc8cfe411749-client-ca-bundle\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.343313 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.343221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de8df011-c523-45a1-8266-dc8cfe411749-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.343313 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.343300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/de8df011-c523-45a1-8266-dc8cfe411749-secret-metrics-server-client-certs\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.444448 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.444398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/de8df011-c523-45a1-8266-dc8cfe411749-secret-metrics-server-tls\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.444915 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.444485    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9c8\" (UniqueName: \"kubernetes.io/projected/de8df011-c523-45a1-8266-dc8cfe411749-kube-api-access-4x9c8\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.444915 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.444511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/de8df011-c523-45a1-8266-dc8cfe411749-audit-log\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.444915 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.444542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/de8df011-c523-45a1-8266-dc8cfe411749-client-ca-bundle\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.444915 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.444578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de8df011-c523-45a1-8266-dc8cfe411749-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.444915 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.444612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/de8df011-c523-45a1-8266-dc8cfe411749-secret-metrics-server-client-certs\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.444915 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.444641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/de8df011-c523-45a1-8266-dc8cfe411749-metrics-server-audit-profiles\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.445237 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.445130    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/de8df011-c523-45a1-8266-dc8cfe411749-audit-log\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.446243 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.445865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de8df011-c523-45a1-8266-dc8cfe411749-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.446391 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.446363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/de8df011-c523-45a1-8266-dc8cfe411749-metrics-server-audit-profiles\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.447977 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.447941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/de8df011-c523-45a1-8266-dc8cfe411749-client-ca-bundle\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.448837 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.448813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/de8df011-c523-45a1-8266-dc8cfe411749-secret-metrics-server-client-certs\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.448945 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.448902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/de8df011-c523-45a1-8266-dc8cfe411749-secret-metrics-server-tls\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.457092 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.457049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9c8\" (UniqueName: \"kubernetes.io/projected/de8df011-c523-45a1-8266-dc8cfe411749-kube-api-access-4x9c8\") pod \"metrics-server-6cc84b76b8-2nclq\" (UID: \"de8df011-c523-45a1-8266-dc8cfe411749\") " pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.544624 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.544586    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"]
Sep 09 20:55:02.550553 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.550520    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:02.553274 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.553245    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ztpj5\""
Sep 09 20:55:02.553451 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.553248    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 20:55:02.557749 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.557161    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"]
Sep 09 20:55:02.588705 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.588670    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:02.646762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.646717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gvc5\" (UniqueName: \"kubernetes.io/projected/d261ebf2-f8d5-4f8f-998f-dafb6640f066-kube-api-access-9gvc5\") pod \"monitoring-plugin-85bbbbbf4-kj57r\" (UID: \"d261ebf2-f8d5-4f8f-998f-dafb6640f066\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:02.647185 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.646829    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d261ebf2-f8d5-4f8f-998f-dafb6640f066-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-kj57r\" (UID: \"d261ebf2-f8d5-4f8f-998f-dafb6640f066\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:02.748065 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.748009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d261ebf2-f8d5-4f8f-998f-dafb6640f066-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-kj57r\" (UID: \"d261ebf2-f8d5-4f8f-998f-dafb6640f066\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:02.748251 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.748139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gvc5\" (UniqueName: \"kubernetes.io/projected/d261ebf2-f8d5-4f8f-998f-dafb6640f066-kube-api-access-9gvc5\") pod \"monitoring-plugin-85bbbbbf4-kj57r\" (UID: \"d261ebf2-f8d5-4f8f-998f-dafb6640f066\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:02.748251 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:55:02.748198    2583 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 20:55:02.748314 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:55:02.748275    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d261ebf2-f8d5-4f8f-998f-dafb6640f066-monitoring-plugin-cert podName:d261ebf2-f8d5-4f8f-998f-dafb6640f066 nodeName:}" failed. No retries permitted until 2026-09-09 20:55:03.248254467 +0000 UTC m=+141.872728402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d261ebf2-f8d5-4f8f-998f-dafb6640f066-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-kj57r" (UID: "d261ebf2-f8d5-4f8f-998f-dafb6640f066") : secret "monitoring-plugin-cert" not found
Sep 09 20:55:02.757860 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.757823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gvc5\" (UniqueName: \"kubernetes.io/projected/d261ebf2-f8d5-4f8f-998f-dafb6640f066-kube-api-access-9gvc5\") pod \"monitoring-plugin-85bbbbbf4-kj57r\" (UID: \"d261ebf2-f8d5-4f8f-998f-dafb6640f066\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:02.877965 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:02.877870    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cc84b76b8-2nclq"]
Sep 09 20:55:03.252437 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:03.252344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d261ebf2-f8d5-4f8f-998f-dafb6640f066-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-kj57r\" (UID: \"d261ebf2-f8d5-4f8f-998f-dafb6640f066\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:03.255300 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:03.255272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d261ebf2-f8d5-4f8f-998f-dafb6640f066-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-kj57r\" (UID: \"d261ebf2-f8d5-4f8f-998f-dafb6640f066\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:03.351008 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:55:03.350960    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8df011_c523_45a1_8266_dc8cfe411749.slice/crio-58ad5855cc0984dedb36f1d77f7b2a3f4907978d7aed84b4e5cdf75befdf6ec1 WatchSource:0}: Error finding container 58ad5855cc0984dedb36f1d77f7b2a3f4907978d7aed84b4e5cdf75befdf6ec1: Status 404 returned error can't find the container with id 58ad5855cc0984dedb36f1d77f7b2a3f4907978d7aed84b4e5cdf75befdf6ec1
Sep 09 20:55:03.451093 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:03.451058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq" event={"ID":"de8df011-c523-45a1-8266-dc8cfe411749","Type":"ContainerStarted","Data":"58ad5855cc0984dedb36f1d77f7b2a3f4907978d7aed84b4e5cdf75befdf6ec1"}
Sep 09 20:55:03.464613 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:03.464582    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:03.784749 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:03.784661    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"]
Sep 09 20:55:03.787671 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:55:03.787622    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd261ebf2_f8d5_4f8f_998f_dafb6640f066.slice/crio-afff35823f96a293eab99dbada966f56d60a43310f5972565a62e31b284795b2 WatchSource:0}: Error finding container afff35823f96a293eab99dbada966f56d60a43310f5972565a62e31b284795b2: Status 404 returned error can't find the container with id afff35823f96a293eab99dbada966f56d60a43310f5972565a62e31b284795b2
Sep 09 20:55:04.045486 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.045445    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:55:04.050270 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.050235    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.055751 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.055604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 20:55:04.055751 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.055635    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 20:55:04.055751 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.055747    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 20:55:04.056048 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.055801    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 20:55:04.056048 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.055927    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-z2c8m\""
Sep 09 20:55:04.057571 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.057541    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 20:55:04.057762 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.057745    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 20:55:04.058071 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.058033    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 20:55:04.059316 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.058313    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 20:55:04.059316 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.058527    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 20:55:04.059501 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-config\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059501 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059501 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059660 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059660 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059660 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059660 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059660 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059885 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059885 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059885 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059752    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059885 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059885 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059885 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059854    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.059885 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059860    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 20:55:04.059885 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.060323 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.060323 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.059973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.060323 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.060000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.060323 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.060056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6lh\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-kube-api-access-dk6lh\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.060323 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.060088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.062538 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.061552    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-94oid6tcvnb1l\""
Sep 09 20:55:04.062538 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.061596    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 20:55:04.062538 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.062064    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 20:55:04.068045 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.068019    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 20:55:04.068883 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.068862    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 20:55:04.081676 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.081642    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:55:04.160972 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.160935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.160972 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.160976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161274 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.160997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161274 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161274 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161274 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161449 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161449 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161449 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6lh\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-kube-api-access-dk6lh\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161449 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161449 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-config\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.161692 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.162239 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161696    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.162239 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.162239 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.161795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.162407 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.162376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.162877 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.162849    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.165263 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.165234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.165546 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.165480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.165955 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.165925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.166056 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.165993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.166642 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.166576    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.166642 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.166578    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.166794 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.166779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.167018 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.166997    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.167557 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.167518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.167557 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.167546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.167703 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.167558    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.167703 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.167577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.168089 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.167968    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.168238 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.168222    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-config\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.168711 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.168469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.170840 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.170787    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6lh\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-kube-api-access-dk6lh\") pod \"prometheus-k8s-0\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.364674 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.364585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:04.460213 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.460173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r" event={"ID":"d261ebf2-f8d5-4f8f-998f-dafb6640f066","Type":"ContainerStarted","Data":"afff35823f96a293eab99dbada966f56d60a43310f5972565a62e31b284795b2"}
Sep 09 20:55:04.465089 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.465046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm" event={"ID":"6508432d-5e5c-4400-83a0-dad1cfe1c39f","Type":"ContainerStarted","Data":"a826eb79f1419c170064bf8761a07e3b673f1e8d4c3e6c3add013b63f2770cc9"}
Sep 09 20:55:04.465268 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.465167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm" event={"ID":"6508432d-5e5c-4400-83a0-dad1cfe1c39f","Type":"ContainerStarted","Data":"25091d849d80dfe7be44fdeff352e8850ab401725ba5abbda3b0cf558639ec86"}
Sep 09 20:55:04.465268 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.465216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm" event={"ID":"6508432d-5e5c-4400-83a0-dad1cfe1c39f","Type":"ContainerStarted","Data":"f541d9b5f3c75364902db921fb774471e435f1a27f577e5be95ad6faf3d42fa6"}
Sep 09 20:55:04.674858 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:04.674778    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:55:04.700540 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:55:04.700501    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da1564b_1938_4e95_92fe_1d2e011b658c.slice/crio-d6d54b55b18579966768de17eb55745bed3d7d908a352ac72bcfcf58383a9e9b WatchSource:0}: Error finding container d6d54b55b18579966768de17eb55745bed3d7d908a352ac72bcfcf58383a9e9b: Status 404 returned error can't find the container with id d6d54b55b18579966768de17eb55745bed3d7d908a352ac72bcfcf58383a9e9b
Sep 09 20:55:05.473136 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:05.473029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm" event={"ID":"6508432d-5e5c-4400-83a0-dad1cfe1c39f","Type":"ContainerStarted","Data":"afc32e926a3aaeddaf37efa17907647bb508d7f4a04d0e8f2698f06a8615e4b6"}
Sep 09 20:55:05.473136 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:05.473130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm" event={"ID":"6508432d-5e5c-4400-83a0-dad1cfe1c39f","Type":"ContainerStarted","Data":"98742dc3b494ab0723ccc5ed4579956683cd56224e8608cc3b632365e1b7dfe7"}
Sep 09 20:55:05.474500 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:05.474474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerStarted","Data":"d6d54b55b18579966768de17eb55745bed3d7d908a352ac72bcfcf58383a9e9b"}
Sep 09 20:55:06.480687 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:06.480644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r" event={"ID":"d261ebf2-f8d5-4f8f-998f-dafb6640f066","Type":"ContainerStarted","Data":"263b5dfec1adad98127486ef26a4b6f0d49a7eb7890d264352d44380f0c34f04"}
Sep 09 20:55:06.481129 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:06.480765    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:06.484791 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:06.484735    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm" event={"ID":"6508432d-5e5c-4400-83a0-dad1cfe1c39f","Type":"ContainerStarted","Data":"ddcf22ee638bb750a78264fc6166ea8e3d50c7d2ccd31440c8d55c6c4360b12c"}
Sep 09 20:55:06.484931 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:06.484884    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:06.486573 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:06.486545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq" event={"ID":"de8df011-c523-45a1-8266-dc8cfe411749","Type":"ContainerStarted","Data":"7cf1830aa0e11c10ab420d241be04faa39dbc7e5d0c55b83effadd614323e03e"}
Sep 09 20:55:06.487560 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:06.487540    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r"
Sep 09 20:55:06.501451 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:06.501391    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kj57r" podStartSLOduration=2.6644880300000002 podStartE2EDuration="4.501375407s" totalImagesPullingTime="1.836887377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:06.501210587 +0000 UTC m=+145.125684527" watchObservedRunningTime="2026-09-09 20:55:06.501375407 +0000 UTC m=+145.125849342"
Sep 09 20:55:06.531513 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:06.531447    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm" podStartSLOduration=3.416579475 podStartE2EDuration="6.531423012s" totalImagesPullingTime="3.114843537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:06.530129293 +0000 UTC m=+145.154603230" watchObservedRunningTime="2026-09-09 20:55:06.531423012 +0000 UTC m=+145.155896955"
Sep 09 20:55:06.550976 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:06.550924    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq" podStartSLOduration=2.275581739 podStartE2EDuration="4.550907389s" totalImagesPullingTime="2.27532565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:06.55040457 +0000 UTC m=+145.174878520" watchObservedRunningTime="2026-09-09 20:55:06.550907389 +0000 UTC m=+145.175381330"
Sep 09 20:55:07.492707 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:07.492669    2583 generic.go:356] "Generic (PLEG): container finished" podID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerID="c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25" exitCode=0
Sep 09 20:55:07.493198 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:07.492748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerDied","Data":"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"}
Sep 09 20:55:10.508827 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:10.508802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerStarted","Data":"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"}
Sep 09 20:55:11.516054 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:11.516018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerStarted","Data":"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"}
Sep 09 20:55:11.516445 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:11.516085    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerStarted","Data":"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"}
Sep 09 20:55:11.516445 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:11.516139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerStarted","Data":"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"}
Sep 09 20:55:11.516445 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:11.516173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerStarted","Data":"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"}
Sep 09 20:55:11.516445 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:11.516206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerStarted","Data":"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"}
Sep 09 20:55:11.564370 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:11.564248    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.662006925 podStartE2EDuration="7.564222753s" totalImagesPullingTime="4.883698904s" totalInitContainerRuntime="18.516924ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:11.562227229 +0000 UTC m=+150.186701171" watchObservedRunningTime="2026-09-09 20:55:11.564222753 +0000 UTC m=+150.188696697"
Sep 09 20:55:12.499376 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:12.499349    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-694bd65f4d-gbcwm"
Sep 09 20:55:14.365620 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:14.365584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:55:19.260815 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:55:19.260767    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rjl4w" podUID="17fc2904-8ec6-4d26-8f39-14abdbe1a1cc"
Sep 09 20:55:19.550342 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:19.550306    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:55:22.588829 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:22.588785    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:22.589227 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:22.588878    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:24.141325 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:24.141288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:55:24.143775 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:24.143755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17fc2904-8ec6-4d26-8f39-14abdbe1a1cc-metrics-tls\") pod \"dns-default-rjl4w\" (UID: \"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc\") " pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:55:24.353874 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:24.353836    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nr885\""
Sep 09 20:55:24.361950 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:24.361921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:55:24.693713 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:24.693634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rjl4w"]
Sep 09 20:55:24.696804 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:55:24.696768    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fc2904_8ec6_4d26_8f39_14abdbe1a1cc.slice/crio-57a99087b9485a0675f3d930259f0f4641c4a771631637eb54960efec77efe72 WatchSource:0}: Error finding container 57a99087b9485a0675f3d930259f0f4641c4a771631637eb54960efec77efe72: Status 404 returned error can't find the container with id 57a99087b9485a0675f3d930259f0f4641c4a771631637eb54960efec77efe72
Sep 09 20:55:25.580499 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:25.580462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rjl4w" event={"ID":"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc","Type":"ContainerStarted","Data":"57a99087b9485a0675f3d930259f0f4641c4a771631637eb54960efec77efe72"}
Sep 09 20:55:27.593349 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:27.593312    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rjl4w" event={"ID":"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc","Type":"ContainerStarted","Data":"27d3e7e7f619c1337199bf9851fe26badd3fe12d81a38c6d1e62cb54b0fd5b79"}
Sep 09 20:55:27.593738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:27.593407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rjl4w" event={"ID":"17fc2904-8ec6-4d26-8f39-14abdbe1a1cc","Type":"ContainerStarted","Data":"6a686cd7feb76a37c3db4530731fe1a41fe829415d6cdfb7827405180d6ad925"}
Sep 09 20:55:27.593738 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:27.593440    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:55:27.613494 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:27.613448    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rjl4w" podStartSLOduration=129.802825798 podStartE2EDuration="2m11.613433618s" totalImagesPullingTime="1.81060782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:27.613285729 +0000 UTC m=+166.237759674" watchObservedRunningTime="2026-09-09 20:55:27.613433618 +0000 UTC m=+166.237907565"
Sep 09 20:55:37.599176 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:37.599070    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rjl4w"
Sep 09 20:55:38.267179 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:38.267143    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rjl4w_17fc2904-8ec6-4d26-8f39-14abdbe1a1cc/dns/0.log"
Sep 09 20:55:38.272693 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:38.272672    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rjl4w_17fc2904-8ec6-4d26-8f39-14abdbe1a1cc/kube-rbac-proxy/0.log"
Sep 09 20:55:38.561834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:38.561809    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mmcfd_b5c2c926-9449-4539-b439-06fb5d910856/dns-node-resolver/0.log"
Sep 09 20:55:42.595926 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:42.594788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:55:42.599632 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:55:42.599605    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6cc84b76b8-2nclq"
Sep 09 20:56:04.365051 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:04.365002    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:04.382140 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:04.381992    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:04.756823 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:04.756751    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:22.414757 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.414715    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:56:22.415188 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.415098    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="prometheus" containerID="cri-o://caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410" gracePeriod=600
Sep 09 20:56:22.415188 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.415163    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="config-reloader" containerID="cri-o://b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a" gracePeriod=600
Sep 09 20:56:22.415320 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.415192    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy-web" containerID="cri-o://39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe" gracePeriod=600
Sep 09 20:56:22.415320 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.415183    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="thanos-sidecar" containerID="cri-o://a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e" gracePeriod=600
Sep 09 20:56:22.415320 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.415145    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy" containerID="cri-o://0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376" gracePeriod=600
Sep 09 20:56:22.415320 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.415182    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a" gracePeriod=600
Sep 09 20:56:22.661123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.661081    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:22.753726 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.753634    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-config-out\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.753726 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.753675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-kube-rbac-proxy\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.753726 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.753703    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-metrics-client-ca\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.753727    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-trusted-ca-bundle\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.753765    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-config\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.753789    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-tls-assets\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.753820    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-thanos-prometheus-http-client-file\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754238 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-web-config\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754238 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754042    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:22.754238 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754084    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-tls\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754238 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754133    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-serving-certs-ca-bundle\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754438 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754416    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:22.754500 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-metrics-client-certs\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754500    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:22.754551 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754519    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-grpc-tls\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754649 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754552    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754649 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754586    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-0\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754649 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754612    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk6lh\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-kube-api-access-dk6lh\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754649 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754645    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-db\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754858 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754676    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-2\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754858 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754700    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754858 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754740    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-1\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.754858 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.754776    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-kubelet-serving-ca-bundle\") pod \"0da1564b-1938-4e95-92fe-1d2e011b658c\" (UID: \"0da1564b-1938-4e95-92fe-1d2e011b658c\") "
Sep 09 20:56:22.755243 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.755032    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.755243 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.755050    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-metrics-client-ca\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.755243 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.755065    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.755389 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.755356    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:22.756052 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.756027    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:22.756928 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.756898    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:22.757121 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.757082    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:22.757373 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.757321    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:22.757579 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.757463    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-config-out" (OuterVolumeSpecName: "config-out") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:22.757722 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.757649    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:22.758322 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.758276    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:22.758769 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.758713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-kube-api-access-dk6lh" (OuterVolumeSpecName: "kube-api-access-dk6lh") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "kube-api-access-dk6lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:22.759226 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.759186    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:22.759347 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.759222    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:22.759668 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.759623    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:22.759668 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.759633    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:22.759803 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.759668    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-config" (OuterVolumeSpecName: "config") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:22.760411 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.760389    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:22.760810 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.760791    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:22.771287 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.771257    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-web-config" (OuterVolumeSpecName: "web-config") pod "0da1564b-1938-4e95-92fe-1d2e011b658c" (UID: "0da1564b-1938-4e95-92fe-1d2e011b658c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:22.815565 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815534    2583 generic.go:356] "Generic (PLEG): container finished" podID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerID="68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a" exitCode=0
Sep 09 20:56:22.815565 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815556    2583 generic.go:356] "Generic (PLEG): container finished" podID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerID="0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376" exitCode=0
Sep 09 20:56:22.815565 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815562    2583 generic.go:356] "Generic (PLEG): container finished" podID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerID="39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe" exitCode=0
Sep 09 20:56:22.815565 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815569    2583 generic.go:356] "Generic (PLEG): container finished" podID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerID="a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e" exitCode=0
Sep 09 20:56:22.815565 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815574    2583 generic.go:356] "Generic (PLEG): container finished" podID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerID="b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a" exitCode=0
Sep 09 20:56:22.815837 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815581    2583 generic.go:356] "Generic (PLEG): container finished" podID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerID="caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410" exitCode=0
Sep 09 20:56:22.815837 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815615    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:22.815837 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815615    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerDied","Data":"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"}
Sep 09 20:56:22.815837 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerDied","Data":"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"}
Sep 09 20:56:22.815837 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815747    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerDied","Data":"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"}
Sep 09 20:56:22.815837 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815766    2583 scope.go:117] "RemoveContainer" containerID="68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"
Sep 09 20:56:22.815837 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815790    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerDied","Data":"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"}
Sep 09 20:56:22.815837 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerDied","Data":"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"}
Sep 09 20:56:22.816125 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerDied","Data":"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"}
Sep 09 20:56:22.816125 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.815917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0da1564b-1938-4e95-92fe-1d2e011b658c","Type":"ContainerDied","Data":"d6d54b55b18579966768de17eb55745bed3d7d908a352ac72bcfcf58383a9e9b"}
Sep 09 20:56:22.825325 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.825305    2583 scope.go:117] "RemoveContainer" containerID="0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"
Sep 09 20:56:22.833419 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.833399    2583 scope.go:117] "RemoveContainer" containerID="39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"
Sep 09 20:56:22.841559 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.841533    2583 scope.go:117] "RemoveContainer" containerID="a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"
Sep 09 20:56:22.845011 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.844987    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:56:22.849734 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.849709    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:56:22.850458 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.850443    2583 scope.go:117] "RemoveContainer" containerID="b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"
Sep 09 20:56:22.855729 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855704    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855729 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855724    2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-metrics-client-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855734    2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-grpc-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855743    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855751    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855760    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6lh\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-kube-api-access-dk6lh\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855768    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-db\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855776    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855784    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855792    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855800    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1564b-1938-4e95-92fe-1d2e011b658c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855808    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0da1564b-1938-4e95-92fe-1d2e011b658c-config-out\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855816    2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-secret-kube-rbac-proxy\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855823    2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855830    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0da1564b-1938-4e95-92fe-1d2e011b658c-tls-assets\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.855834 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855838    2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.856294 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.855846    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0da1564b-1938-4e95-92fe-1d2e011b658c-web-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 20:56:22.857628 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.857613    2583 scope.go:117] "RemoveContainer" containerID="caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"
Sep 09 20:56:22.864868 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.864851    2583 scope.go:117] "RemoveContainer" containerID="c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"
Sep 09 20:56:22.871821 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.871804    2583 scope.go:117] "RemoveContainer" containerID="68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"
Sep 09 20:56:22.872140 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:56:22.872117    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": container with ID starting with 68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a not found: ID does not exist" containerID="68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"
Sep 09 20:56:22.872198 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.872151    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"} err="failed to get container status \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": rpc error: code = NotFound desc = could not find container \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": container with ID starting with 68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a not found: ID does not exist"
Sep 09 20:56:22.872198 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.872184    2583 scope.go:117] "RemoveContainer" containerID="0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"
Sep 09 20:56:22.872447 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:56:22.872429    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": container with ID starting with 0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376 not found: ID does not exist" containerID="0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"
Sep 09 20:56:22.872489 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.872454    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"} err="failed to get container status \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": rpc error: code = NotFound desc = could not find container \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": container with ID starting with 0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376 not found: ID does not exist"
Sep 09 20:56:22.872489 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.872469    2583 scope.go:117] "RemoveContainer" containerID="39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"
Sep 09 20:56:22.872714 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:56:22.872682    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": container with ID starting with 39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe not found: ID does not exist" containerID="39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"
Sep 09 20:56:22.872873 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.872711    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"} err="failed to get container status \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": rpc error: code = NotFound desc = could not find container \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": container with ID starting with 39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe not found: ID does not exist"
Sep 09 20:56:22.872873 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.872734    2583 scope.go:117] "RemoveContainer" containerID="a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"
Sep 09 20:56:22.873194 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:56:22.873033    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": container with ID starting with a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e not found: ID does not exist" containerID="a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"
Sep 09 20:56:22.873194 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.873057    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"} err="failed to get container status \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": rpc error: code = NotFound desc = could not find container \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": container with ID starting with a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e not found: ID does not exist"
Sep 09 20:56:22.873194 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.873091    2583 scope.go:117] "RemoveContainer" containerID="b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"
Sep 09 20:56:22.873443 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:56:22.873426    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": container with ID starting with b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a not found: ID does not exist" containerID="b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"
Sep 09 20:56:22.873482 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.873447    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"} err="failed to get container status \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": rpc error: code = NotFound desc = could not find container \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": container with ID starting with b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a not found: ID does not exist"
Sep 09 20:56:22.873482 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.873460    2583 scope.go:117] "RemoveContainer" containerID="caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"
Sep 09 20:56:22.873697 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:56:22.873682    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": container with ID starting with caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410 not found: ID does not exist" containerID="caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"
Sep 09 20:56:22.873756 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.873703    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"} err="failed to get container status \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": rpc error: code = NotFound desc = could not find container \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": container with ID starting with caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410 not found: ID does not exist"
Sep 09 20:56:22.873756 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.873717    2583 scope.go:117] "RemoveContainer" containerID="c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"
Sep 09 20:56:22.873973 ip-10-0-140-132 kubenswrapper[2583]: E0909 20:56:22.873954    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": container with ID starting with c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25 not found: ID does not exist" containerID="c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"
Sep 09 20:56:22.874036 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.873977    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"} err="failed to get container status \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": rpc error: code = NotFound desc = could not find container \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": container with ID starting with c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25 not found: ID does not exist"
Sep 09 20:56:22.874036 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.873991    2583 scope.go:117] "RemoveContainer" containerID="68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"
Sep 09 20:56:22.874259 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.874237    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"} err="failed to get container status \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": rpc error: code = NotFound desc = could not find container \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": container with ID starting with 68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a not found: ID does not exist"
Sep 09 20:56:22.874317 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.874265    2583 scope.go:117] "RemoveContainer" containerID="0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"
Sep 09 20:56:22.874523 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.874506    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"} err="failed to get container status \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": rpc error: code = NotFound desc = could not find container \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": container with ID starting with 0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376 not found: ID does not exist"
Sep 09 20:56:22.874523 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.874522    2583 scope.go:117] "RemoveContainer" containerID="39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"
Sep 09 20:56:22.874746 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.874731    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"} err="failed to get container status \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": rpc error: code = NotFound desc = could not find container \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": container with ID starting with 39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe not found: ID does not exist"
Sep 09 20:56:22.874793 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.874746    2583 scope.go:117] "RemoveContainer" containerID="a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"
Sep 09 20:56:22.874984 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.874966    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"} err="failed to get container status \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": rpc error: code = NotFound desc = could not find container \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": container with ID starting with a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e not found: ID does not exist"
Sep 09 20:56:22.875029 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.874985    2583 scope.go:117] "RemoveContainer" containerID="b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"
Sep 09 20:56:22.875225 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.875210    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"} err="failed to get container status \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": rpc error: code = NotFound desc = could not find container \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": container with ID starting with b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a not found: ID does not exist"
Sep 09 20:56:22.875277 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.875225    2583 scope.go:117] "RemoveContainer" containerID="caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"
Sep 09 20:56:22.875450 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.875435    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"} err="failed to get container status \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": rpc error: code = NotFound desc = could not find container \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": container with ID starting with caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410 not found: ID does not exist"
Sep 09 20:56:22.875450 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.875449    2583 scope.go:117] "RemoveContainer" containerID="c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"
Sep 09 20:56:22.875631 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.875616    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"} err="failed to get container status \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": rpc error: code = NotFound desc = could not find container \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": container with ID starting with c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25 not found: ID does not exist"
Sep 09 20:56:22.875673 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.875631    2583 scope.go:117] "RemoveContainer" containerID="68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"
Sep 09 20:56:22.875821 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.875807    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"} err="failed to get container status \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": rpc error: code = NotFound desc = could not find container \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": container with ID starting with 68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a not found: ID does not exist"
Sep 09 20:56:22.875866 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.875823    2583 scope.go:117] "RemoveContainer" containerID="0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"
Sep 09 20:56:22.876034 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876019    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"} err="failed to get container status \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": rpc error: code = NotFound desc = could not find container \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": container with ID starting with 0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376 not found: ID does not exist"
Sep 09 20:56:22.876034 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876033    2583 scope.go:117] "RemoveContainer" containerID="39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"
Sep 09 20:56:22.876240 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876223    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"} err="failed to get container status \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": rpc error: code = NotFound desc = could not find container \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": container with ID starting with 39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe not found: ID does not exist"
Sep 09 20:56:22.876290 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876240    2583 scope.go:117] "RemoveContainer" containerID="a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"
Sep 09 20:56:22.876463 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876446    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"} err="failed to get container status \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": rpc error: code = NotFound desc = could not find container \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": container with ID starting with a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e not found: ID does not exist"
Sep 09 20:56:22.876514 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876464    2583 scope.go:117] "RemoveContainer" containerID="b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"
Sep 09 20:56:22.876657 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876642    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"} err="failed to get container status \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": rpc error: code = NotFound desc = could not find container \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": container with ID starting with b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a not found: ID does not exist"
Sep 09 20:56:22.876699 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876657    2583 scope.go:117] "RemoveContainer" containerID="caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"
Sep 09 20:56:22.876863 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876849    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"} err="failed to get container status \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": rpc error: code = NotFound desc = could not find container \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": container with ID starting with caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410 not found: ID does not exist"
Sep 09 20:56:22.876905 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.876863    2583 scope.go:117] "RemoveContainer" containerID="c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"
Sep 09 20:56:22.877036 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877021    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"} err="failed to get container status \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": rpc error: code = NotFound desc = could not find container \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": container with ID starting with c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25 not found: ID does not exist"
Sep 09 20:56:22.877077 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877035    2583 scope.go:117] "RemoveContainer" containerID="68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"
Sep 09 20:56:22.877298 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877281    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"} err="failed to get container status \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": rpc error: code = NotFound desc = could not find container \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": container with ID starting with 68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a not found: ID does not exist"
Sep 09 20:56:22.877364 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877299    2583 scope.go:117] "RemoveContainer" containerID="0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"
Sep 09 20:56:22.877532 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877514    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"} err="failed to get container status \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": rpc error: code = NotFound desc = could not find container \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": container with ID starting with 0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376 not found: ID does not exist"
Sep 09 20:56:22.877573 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877541    2583 scope.go:117] "RemoveContainer" containerID="39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"
Sep 09 20:56:22.877751 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877736    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"} err="failed to get container status \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": rpc error: code = NotFound desc = could not find container \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": container with ID starting with 39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe not found: ID does not exist"
Sep 09 20:56:22.877794 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877751    2583 scope.go:117] "RemoveContainer" containerID="a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"
Sep 09 20:56:22.877969 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877954    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"} err="failed to get container status \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": rpc error: code = NotFound desc = could not find container \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": container with ID starting with a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e not found: ID does not exist"
Sep 09 20:56:22.877969 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.877968    2583 scope.go:117] "RemoveContainer" containerID="b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"
Sep 09 20:56:22.878200 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.878184    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"} err="failed to get container status \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": rpc error: code = NotFound desc = could not find container \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": container with ID starting with b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a not found: ID does not exist"
Sep 09 20:56:22.878245 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.878199    2583 scope.go:117] "RemoveContainer" containerID="caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"
Sep 09 20:56:22.878412 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.878396    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"} err="failed to get container status \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": rpc error: code = NotFound desc = could not find container \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": container with ID starting with caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410 not found: ID does not exist"
Sep 09 20:56:22.878412 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.878411    2583 scope.go:117] "RemoveContainer" containerID="c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"
Sep 09 20:56:22.878609 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.878593    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"} err="failed to get container status \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": rpc error: code = NotFound desc = could not find container \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": container with ID starting with c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25 not found: ID does not exist"
Sep 09 20:56:22.878649 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.878609    2583 scope.go:117] "RemoveContainer" containerID="68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"
Sep 09 20:56:22.878841 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.878826    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"} err="failed to get container status \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": rpc error: code = NotFound desc = could not find container \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": container with ID starting with 68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a not found: ID does not exist"
Sep 09 20:56:22.878841 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.878840    2583 scope.go:117] "RemoveContainer" containerID="0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"
Sep 09 20:56:22.879042 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879028    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"} err="failed to get container status \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": rpc error: code = NotFound desc = could not find container \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": container with ID starting with 0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376 not found: ID does not exist"
Sep 09 20:56:22.879042 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879041    2583 scope.go:117] "RemoveContainer" containerID="39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"
Sep 09 20:56:22.879273 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879256    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"} err="failed to get container status \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": rpc error: code = NotFound desc = could not find container \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": container with ID starting with 39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe not found: ID does not exist"
Sep 09 20:56:22.879273 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879273    2583 scope.go:117] "RemoveContainer" containerID="a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"
Sep 09 20:56:22.879485 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879469    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"} err="failed to get container status \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": rpc error: code = NotFound desc = could not find container \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": container with ID starting with a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e not found: ID does not exist"
Sep 09 20:56:22.879485 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879484    2583 scope.go:117] "RemoveContainer" containerID="b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"
Sep 09 20:56:22.879690 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879676    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"} err="failed to get container status \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": rpc error: code = NotFound desc = could not find container \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": container with ID starting with b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a not found: ID does not exist"
Sep 09 20:56:22.879734 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879690    2583 scope.go:117] "RemoveContainer" containerID="caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"
Sep 09 20:56:22.879898 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879883    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"} err="failed to get container status \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": rpc error: code = NotFound desc = could not find container \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": container with ID starting with caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410 not found: ID does not exist"
Sep 09 20:56:22.879898 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.879897    2583 scope.go:117] "RemoveContainer" containerID="c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"
Sep 09 20:56:22.880095 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880082    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"} err="failed to get container status \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": rpc error: code = NotFound desc = could not find container \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": container with ID starting with c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25 not found: ID does not exist"
Sep 09 20:56:22.880095 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880094    2583 scope.go:117] "RemoveContainer" containerID="68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"
Sep 09 20:56:22.880338 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880321    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a"} err="failed to get container status \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": rpc error: code = NotFound desc = could not find container \"68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a\": container with ID starting with 68a56655020ae790ca900259b30fc57b2c74b370a586b707e85200cdab85935a not found: ID does not exist"
Sep 09 20:56:22.880338 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880336    2583 scope.go:117] "RemoveContainer" containerID="0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"
Sep 09 20:56:22.880541 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880527    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376"} err="failed to get container status \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": rpc error: code = NotFound desc = could not find container \"0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376\": container with ID starting with 0df83098cffed474a114c165c48b6771384d53ff0c399a61d72f8b688c4f8376 not found: ID does not exist"
Sep 09 20:56:22.880581 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880541    2583 scope.go:117] "RemoveContainer" containerID="39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"
Sep 09 20:56:22.880748 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880732    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe"} err="failed to get container status \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": rpc error: code = NotFound desc = could not find container \"39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe\": container with ID starting with 39c6c13dff1f8db04fed4880d80c5cf7b152ba30423092a64afa7100fb02febe not found: ID does not exist"
Sep 09 20:56:22.880748 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880746    2583 scope.go:117] "RemoveContainer" containerID="a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"
Sep 09 20:56:22.880954 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880931    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e"} err="failed to get container status \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": rpc error: code = NotFound desc = could not find container \"a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e\": container with ID starting with a6cc5edd6fc51a4308ddd36c4208fdb667daa7b87e061c06fdb33197f3e1272e not found: ID does not exist"
Sep 09 20:56:22.881012 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.880954    2583 scope.go:117] "RemoveContainer" containerID="b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"
Sep 09 20:56:22.881563 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.881387    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a"} err="failed to get container status \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": rpc error: code = NotFound desc = could not find container \"b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a\": container with ID starting with b3e202613014c0e35415276065d3fffeac07ad8cce2041652116c55776b42d1a not found: ID does not exist"
Sep 09 20:56:22.881563 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.881410    2583 scope.go:117] "RemoveContainer" containerID="caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"
Sep 09 20:56:22.882026 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.881785    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410"} err="failed to get container status \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": rpc error: code = NotFound desc = could not find container \"caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410\": container with ID starting with caf078b62c9db5f5f4c4a1c67ae8c821d2ac24b62356fd34a895b86236753410 not found: ID does not exist"
Sep 09 20:56:22.882026 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.881812    2583 scope.go:117] "RemoveContainer" containerID="c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"
Sep 09 20:56:22.882205 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.882093    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25"} err="failed to get container status \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": rpc error: code = NotFound desc = could not find container \"c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25\": container with ID starting with c27ec93014c53cf9c83df86d8105eab64aff8ece6af4ee5e12f36d71f70a2a25 not found: ID does not exist"
Sep 09 20:56:22.883081 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883064    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:56:22.883367 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883354    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy"
Sep 09 20:56:22.883408 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883368    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy"
Sep 09 20:56:22.883408 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883378    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy-thanos"
Sep 09 20:56:22.883408 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883384    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy-thanos"
Sep 09 20:56:22.883408 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883393    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="init-config-reloader"
Sep 09 20:56:22.883408 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883398    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="init-config-reloader"
Sep 09 20:56:22.883408 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883405    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="prometheus"
Sep 09 20:56:22.883408 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883410    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="prometheus"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883421    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="thanos-sidecar"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883425    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="thanos-sidecar"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883431    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy-web"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883436    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy-web"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883444    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="config-reloader"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883449    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="config-reloader"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883490    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="config-reloader"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883498    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883504    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="thanos-sidecar"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883511    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy-web"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883516    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="kube-rbac-proxy-thanos"
Sep 09 20:56:22.883598 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.883521    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" containerName="prometheus"
Sep 09 20:56:22.888643 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.888626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:22.892020 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.892005    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 20:56:22.892223 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.892207    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 20:56:22.892436 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.892419    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 20:56:22.892562 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.892549    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 20:56:22.893193 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.893178    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 20:56:22.895198 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.895176    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 20:56:22.895442 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.895419    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 20:56:22.896354 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.895569    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 20:56:22.896354 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.895579    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-94oid6tcvnb1l\""
Sep 09 20:56:22.896354 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.895792    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-z2c8m\""
Sep 09 20:56:22.896354 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.895815    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 20:56:22.896354 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.895864    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 20:56:22.896354 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.895955    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 20:56:22.896354 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.896007    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 20:56:22.896354 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.896255    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 20:56:22.905889 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.904738    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 20:56:22.910980 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:22.910954    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:56:23.058228 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058390 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058390 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058390 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058486 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058486 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058486 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058486 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058469    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058601 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058601 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058601 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058601 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058718 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b93d5670-d259-4d7d-80f5-0c96dced55c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058718 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058718 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45hv\" (UniqueName: \"kubernetes.io/projected/b93d5670-d259-4d7d-80f5-0c96dced55c3-kube-api-access-t45hv\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058718 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-config\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058718 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058852 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b93d5670-d259-4d7d-80f5-0c96dced55c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058852 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.058852 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.058787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.159895 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.159857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.159895 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.159897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b93d5670-d259-4d7d-80f5-0c96dced55c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.160130 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.159913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.160130 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.160055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45hv\" (UniqueName: \"kubernetes.io/projected/b93d5670-d259-4d7d-80f5-0c96dced55c3-kube-api-access-t45hv\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.160130 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.160095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-config\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.160301 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.160141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.160301 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.160168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b93d5670-d259-4d7d-80f5-0c96dced55c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.160301 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.160196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.160301 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.160224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.160428 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.160299    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.160428 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.160315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161123 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161072    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161231 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161231 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161231 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161402 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161402 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161402 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161402 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161402 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161402 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161402 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161727 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.161973 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.161945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.163453 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.163272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b93d5670-d259-4d7d-80f5-0c96dced55c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.163453 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.163400    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-config\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.164078 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.163759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.164078 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.163890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.164078 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.164012    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b93d5670-d259-4d7d-80f5-0c96dced55c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.164078 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.164027    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.164411 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.164386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.164720 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.164658    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.164720 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.164734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.165003 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.164977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93d5670-d259-4d7d-80f5-0c96dced55c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.165274 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.165250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.165691 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.165675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.166389 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.166365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.166758 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.166731    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.167008 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.166992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b93d5670-d259-4d7d-80f5-0c96dced55c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.170007 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.169988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45hv\" (UniqueName: \"kubernetes.io/projected/b93d5670-d259-4d7d-80f5-0c96dced55c3-kube-api-access-t45hv\") pod \"prometheus-k8s-0\" (UID: \"b93d5670-d259-4d7d-80f5-0c96dced55c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.199191 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.199152    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:56:23.492154 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.492128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:56:23.493817 ip-10-0-140-132 kubenswrapper[2583]: W0909 20:56:23.493791    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93d5670_d259_4d7d_80f5_0c96dced55c3.slice/crio-eadbfa1e55c0d895dfb23a64b0996709685c4987a55e75c47ba698b0ffb0c9ab WatchSource:0}: Error finding container eadbfa1e55c0d895dfb23a64b0996709685c4987a55e75c47ba698b0ffb0c9ab: Status 404 returned error can't find the container with id eadbfa1e55c0d895dfb23a64b0996709685c4987a55e75c47ba698b0ffb0c9ab
Sep 09 20:56:23.820619 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.820583    2583 generic.go:356] "Generic (PLEG): container finished" podID="b93d5670-d259-4d7d-80f5-0c96dced55c3" containerID="f2f8b99c100db2bdc639350e56397bf2f984b9f3c419bf563308a88ab188a8de" exitCode=0
Sep 09 20:56:23.820795 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.820680    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93d5670-d259-4d7d-80f5-0c96dced55c3","Type":"ContainerDied","Data":"f2f8b99c100db2bdc639350e56397bf2f984b9f3c419bf563308a88ab188a8de"}
Sep 09 20:56:23.820795 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.820749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93d5670-d259-4d7d-80f5-0c96dced55c3","Type":"ContainerStarted","Data":"eadbfa1e55c0d895dfb23a64b0996709685c4987a55e75c47ba698b0ffb0c9ab"}
Sep 09 20:56:23.865545 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:23.865516    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da1564b-1938-4e95-92fe-1d2e011b658c" path="/var/lib/kubelet/pods/0da1564b-1938-4e95-92fe-1d2e011b658c/volumes"
Sep 09 20:56:24.828983 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:24.828948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93d5670-d259-4d7d-80f5-0c96dced55c3","Type":"ContainerStarted","Data":"a8938f08d598676316f22ddc73b2b7f60dfc3ddc2a90decca37aa6ad64ddd5ea"}
Sep 09 20:56:24.829452 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:24.829010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93d5670-d259-4d7d-80f5-0c96dced55c3","Type":"ContainerStarted","Data":"791ac15becd542c302f0abb2b509daf5c5c19236faaed61829b6bd3503dc1208"}
Sep 09 20:56:24.829452 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:24.829039    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93d5670-d259-4d7d-80f5-0c96dced55c3","Type":"ContainerStarted","Data":"f02298af383bf162bb6b9e40ae52262829d8a27d658fa766608810e2cbda5c62"}
Sep 09 20:56:24.829452 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:24.829062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93d5670-d259-4d7d-80f5-0c96dced55c3","Type":"ContainerStarted","Data":"e4f3fd2ba8d95c7ee6d02d0a88b2b046186f21ce42f1494df9f3d7522a42a340"}
Sep 09 20:56:24.829452 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:24.829087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93d5670-d259-4d7d-80f5-0c96dced55c3","Type":"ContainerStarted","Data":"5c48bdc64b750f8ed95af6bc03ab6fce667a8d8cde4ebd2512ad2249b6ea338e"}
Sep 09 20:56:24.829452 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:24.829136    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93d5670-d259-4d7d-80f5-0c96dced55c3","Type":"ContainerStarted","Data":"59c7421d79a77d741c46e2303da4cab35806cff2a113c374d8af75a094fa9879"}
Sep 09 20:56:24.861836 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:24.861783    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.842007464 podStartE2EDuration="2.861766254s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.75879ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:24.859671418 +0000 UTC m=+223.484145360" watchObservedRunningTime="2026-09-09 20:56:24.861766254 +0000 UTC m=+223.486240194"
Sep 09 20:56:28.200299 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:56:28.200259    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:57:23.200256 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:57:23.200221    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:57:23.216952 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:57:23.216926    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:57:24.076063 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:57:24.076036    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:57:41.811304 ip-10-0-140-132 kubenswrapper[2583]: I0909 20:57:41.811273    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 21:00:32.745026 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.744993    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"]
Sep 09 21:00:32.748516 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.748498    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"
Sep 09 21:00:32.751942 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.751922    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:00:32.752063 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.751963    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7z6ph\""
Sep 09 21:00:32.752977 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.752960    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 21:00:32.757448 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.757423    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"]
Sep 09 21:00:32.828405 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.828366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e340f7b6-75b1-4796-80f8-699db99dc8f8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9bdfv\" (UID: \"e340f7b6-75b1-4796-80f8-699db99dc8f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"
Sep 09 21:00:32.828405 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.828408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzq7\" (UniqueName: \"kubernetes.io/projected/e340f7b6-75b1-4796-80f8-699db99dc8f8-kube-api-access-8kzq7\") pod \"openshift-lws-operator-fd8ccff4c-9bdfv\" (UID: \"e340f7b6-75b1-4796-80f8-699db99dc8f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"
Sep 09 21:00:32.928931 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.928889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e340f7b6-75b1-4796-80f8-699db99dc8f8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9bdfv\" (UID: \"e340f7b6-75b1-4796-80f8-699db99dc8f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"
Sep 09 21:00:32.929079 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.928956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzq7\" (UniqueName: \"kubernetes.io/projected/e340f7b6-75b1-4796-80f8-699db99dc8f8-kube-api-access-8kzq7\") pod \"openshift-lws-operator-fd8ccff4c-9bdfv\" (UID: \"e340f7b6-75b1-4796-80f8-699db99dc8f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"
Sep 09 21:00:32.929266 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.929245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e340f7b6-75b1-4796-80f8-699db99dc8f8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9bdfv\" (UID: \"e340f7b6-75b1-4796-80f8-699db99dc8f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"
Sep 09 21:00:32.938911 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:32.938883    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzq7\" (UniqueName: \"kubernetes.io/projected/e340f7b6-75b1-4796-80f8-699db99dc8f8-kube-api-access-8kzq7\") pod \"openshift-lws-operator-fd8ccff4c-9bdfv\" (UID: \"e340f7b6-75b1-4796-80f8-699db99dc8f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"
Sep 09 21:00:33.058032 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:33.058001    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"
Sep 09 21:00:33.339548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:33.339523    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv"]
Sep 09 21:00:33.342156 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:00:33.342127    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode340f7b6_75b1_4796_80f8_699db99dc8f8.slice/crio-afa1b2bffcbcf53075e8851901615d77660550525fa937375cf457641aeef4c9 WatchSource:0}: Error finding container afa1b2bffcbcf53075e8851901615d77660550525fa937375cf457641aeef4c9: Status 404 returned error can't find the container with id afa1b2bffcbcf53075e8851901615d77660550525fa937375cf457641aeef4c9
Sep 09 21:00:33.343837 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:33.343815    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:00:33.800856 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:33.800816    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv" event={"ID":"e340f7b6-75b1-4796-80f8-699db99dc8f8","Type":"ContainerStarted","Data":"afa1b2bffcbcf53075e8851901615d77660550525fa937375cf457641aeef4c9"}
Sep 09 21:00:35.811648 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:35.811615    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv" event={"ID":"e340f7b6-75b1-4796-80f8-699db99dc8f8","Type":"ContainerStarted","Data":"b16cc4328bae20c7611ed469ed211bbe91d8c612e4319069750de8bb8bca06a2"}
Sep 09 21:00:35.831261 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:35.831209    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9bdfv" podStartSLOduration=1.539036031 podStartE2EDuration="3.831192827s" totalImagesPullingTime="2.292156796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:35.829234476 +0000 UTC m=+474.453708417" watchObservedRunningTime="2026-09-09 21:00:35.831192827 +0000 UTC m=+474.455666767"
Sep 09 21:00:50.585786 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.585748    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"]
Sep 09 21:00:50.589353 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.589334    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.593650 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.593621    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6kknb\""
Sep 09 21:00:50.593794 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.593650    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 21:00:50.593794 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.593674    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 21:00:50.593794 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.593695    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 21:00:50.607766 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.607730    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"]
Sep 09 21:00:50.693497 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.693459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59bb1174-2459-4493-bfdc-44f6232ce83d-metrics-cert\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.693671 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.693507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/59bb1174-2459-4493-bfdc-44f6232ce83d-manager-config\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.693671 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.693559    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbjj\" (UniqueName: \"kubernetes.io/projected/59bb1174-2459-4493-bfdc-44f6232ce83d-kube-api-access-jcbjj\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.693671 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.693580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59bb1174-2459-4493-bfdc-44f6232ce83d-cert\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.794413 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.794372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59bb1174-2459-4493-bfdc-44f6232ce83d-cert\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.794615 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.794483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59bb1174-2459-4493-bfdc-44f6232ce83d-metrics-cert\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.794615 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.794546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/59bb1174-2459-4493-bfdc-44f6232ce83d-manager-config\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.794615 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.794596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbjj\" (UniqueName: \"kubernetes.io/projected/59bb1174-2459-4493-bfdc-44f6232ce83d-kube-api-access-jcbjj\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.795293 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.795214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/59bb1174-2459-4493-bfdc-44f6232ce83d-manager-config\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.797324 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.797294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59bb1174-2459-4493-bfdc-44f6232ce83d-metrics-cert\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.797460 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.797360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59bb1174-2459-4493-bfdc-44f6232ce83d-cert\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.814829 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.814792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbjj\" (UniqueName: \"kubernetes.io/projected/59bb1174-2459-4493-bfdc-44f6232ce83d-kube-api-access-jcbjj\") pod \"lws-controller-manager-85b69767b4-7dc9x\" (UID: \"59bb1174-2459-4493-bfdc-44f6232ce83d\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:50.898946 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:50.898856    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:51.172084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:51.172057    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"]
Sep 09 21:00:51.174615 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:00:51.174586    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bb1174_2459_4493_bfdc_44f6232ce83d.slice/crio-828b16d9ac5fc39c7f98aa6ee367e4411a4888867b54e33d01d27e10220b48ea WatchSource:0}: Error finding container 828b16d9ac5fc39c7f98aa6ee367e4411a4888867b54e33d01d27e10220b48ea: Status 404 returned error can't find the container with id 828b16d9ac5fc39c7f98aa6ee367e4411a4888867b54e33d01d27e10220b48ea
Sep 09 21:00:51.881915 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:51.881860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x" event={"ID":"59bb1174-2459-4493-bfdc-44f6232ce83d","Type":"ContainerStarted","Data":"828b16d9ac5fc39c7f98aa6ee367e4411a4888867b54e33d01d27e10220b48ea"}
Sep 09 21:00:52.887295 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:52.887257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x" event={"ID":"59bb1174-2459-4493-bfdc-44f6232ce83d","Type":"ContainerStarted","Data":"06a280d64bb2d54cfb9e0445afc5f42e995bdb4638ad4e94b55ac6cb3a08e275"}
Sep 09 21:00:52.887770 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:52.887390    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:00:52.912695 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:52.912650    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x" podStartSLOduration=1.669769242 podStartE2EDuration="2.912634554s" totalImagesPullingTime="1.242865312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:52.911235859 +0000 UTC m=+491.535709800" watchObservedRunningTime="2026-09-09 21:00:52.912634554 +0000 UTC m=+491.537108495"
Sep 09 21:00:53.713688 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.713657    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"]
Sep 09 21:00:53.717731 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.717710    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.720939 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.720919    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:00:53.721811 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.721793    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-cts8w\""
Sep 09 21:00:53.758239 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.758206    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"]
Sep 09 21:00:53.821723 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.821688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d138945c-aa5e-4fb7-86c5-48e959a29350-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.821920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.821743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-data\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.821920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.821762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-token\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.821920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.821830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-workload-certs\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.821920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.821866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-envoy\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.821920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.821890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sff95\" (UniqueName: \"kubernetes.io/projected/d138945c-aa5e-4fb7-86c5-48e959a29350-kube-api-access-sff95\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.822141 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.821929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-credential-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.822141 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.821976    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.822141 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.822008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-workload-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923089 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-credential-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923089 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923578 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-workload-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923578 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d138945c-aa5e-4fb7-86c5-48e959a29350-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923578 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-data\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923578 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-token\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923578 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-workload-certs\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923578 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-envoy\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923578 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sff95\" (UniqueName: \"kubernetes.io/projected/d138945c-aa5e-4fb7-86c5-48e959a29350-kube-api-access-sff95\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923578 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-credential-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923578 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-workload-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.923985 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-data\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.924474 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.924446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-workload-certs\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.924588 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.923943    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d138945c-aa5e-4fb7-86c5-48e959a29350-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.926222 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.926178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-envoy\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.926520 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.926483    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.937778 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.937756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d138945c-aa5e-4fb7-86c5-48e959a29350-istio-token\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:53.938738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:53.938719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sff95\" (UniqueName: \"kubernetes.io/projected/d138945c-aa5e-4fb7-86c5-48e959a29350-kube-api-access-sff95\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k\" (UID: \"d138945c-aa5e-4fb7-86c5-48e959a29350\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:54.029203 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:54.029119    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:54.312847 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:54.312820    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"]
Sep 09 21:00:54.315349 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:00:54.315311    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd138945c_aa5e_4fb7_86c5_48e959a29350.slice/crio-7d5c9418be0b6660f789dddb882b63a0a09b7fe2040b26ab71b204ccd2096c48 WatchSource:0}: Error finding container 7d5c9418be0b6660f789dddb882b63a0a09b7fe2040b26ab71b204ccd2096c48: Status 404 returned error can't find the container with id 7d5c9418be0b6660f789dddb882b63a0a09b7fe2040b26ab71b204ccd2096c48
Sep 09 21:00:54.897216 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:54.897161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" event={"ID":"d138945c-aa5e-4fb7-86c5-48e959a29350","Type":"ContainerStarted","Data":"7d5c9418be0b6660f789dddb882b63a0a09b7fe2040b26ab71b204ccd2096c48"}
Sep 09 21:00:56.803931 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:56.803886    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:00:56.804326 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:56.803980    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:00:56.804326 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:56.804029    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:00:56.908603 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:56.908561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" event={"ID":"d138945c-aa5e-4fb7-86c5-48e959a29350","Type":"ContainerStarted","Data":"69b936e399f7728f9187fe0dc7711d4694c510c443c7c0f1a9f552b4a5c6ff0b"}
Sep 09 21:00:56.935655 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:56.935600    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" podStartSLOduration=1.449411393 podStartE2EDuration="3.935583491s" totalImagesPullingTime="2.486172098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:56.933617567 +0000 UTC m=+495.558091510" watchObservedRunningTime="2026-09-09 21:00:56.935583491 +0000 UTC m=+495.560057432"
Sep 09 21:00:57.029920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:57.029872    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:57.031522 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:57.031490    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 09 21:00:57.031637 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:57.031548    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" podUID="d138945c-aa5e-4fb7-86c5-48e959a29350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 09 21:00:58.033964 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:58.033931    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:58.915877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:58.915848    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:00:58.916825 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:00:58.916799    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:01:03.893724 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:03.893691    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-7dc9x"
Sep 09 21:01:27.159493 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.159460    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"]
Sep 09 21:01:27.169225 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.169205    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:01:27.172240 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.172221    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wzstv\""
Sep 09 21:01:27.172461 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.172446    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 21:01:27.172504 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.172446    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 21:01:27.179040 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.179020    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"]
Sep 09 21:01:27.237697 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.237657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff3d4ace-6b31-4357-98ac-255b15446a5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dzn25\" (UID: \"ff3d4ace-6b31-4357-98ac-255b15446a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:01:27.237895 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.237775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnjf\" (UniqueName: \"kubernetes.io/projected/ff3d4ace-6b31-4357-98ac-255b15446a5e-kube-api-access-gfnjf\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dzn25\" (UID: \"ff3d4ace-6b31-4357-98ac-255b15446a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:01:27.338840 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.338791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnjf\" (UniqueName: \"kubernetes.io/projected/ff3d4ace-6b31-4357-98ac-255b15446a5e-kube-api-access-gfnjf\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dzn25\" (UID: \"ff3d4ace-6b31-4357-98ac-255b15446a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:01:27.338840 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.338855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff3d4ace-6b31-4357-98ac-255b15446a5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dzn25\" (UID: \"ff3d4ace-6b31-4357-98ac-255b15446a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:01:27.339234 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.339214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff3d4ace-6b31-4357-98ac-255b15446a5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dzn25\" (UID: \"ff3d4ace-6b31-4357-98ac-255b15446a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:01:27.354172 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.354145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnjf\" (UniqueName: \"kubernetes.io/projected/ff3d4ace-6b31-4357-98ac-255b15446a5e-kube-api-access-gfnjf\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dzn25\" (UID: \"ff3d4ace-6b31-4357-98ac-255b15446a5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:01:27.479641 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.479551    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:01:27.759520 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:27.759467    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"]
Sep 09 21:01:27.762295 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:01:27.762266    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3d4ace_6b31_4357_98ac_255b15446a5e.slice/crio-1c4a1b0127478232a3b915d4bc76e212840ae38d7c083de3404ddf6bdf2bc65b WatchSource:0}: Error finding container 1c4a1b0127478232a3b915d4bc76e212840ae38d7c083de3404ddf6bdf2bc65b: Status 404 returned error can't find the container with id 1c4a1b0127478232a3b915d4bc76e212840ae38d7c083de3404ddf6bdf2bc65b
Sep 09 21:01:28.034890 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:28.034796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25" event={"ID":"ff3d4ace-6b31-4357-98ac-255b15446a5e","Type":"ContainerStarted","Data":"1c4a1b0127478232a3b915d4bc76e212840ae38d7c083de3404ddf6bdf2bc65b"}
Sep 09 21:01:32.058150 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:32.058097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25" event={"ID":"ff3d4ace-6b31-4357-98ac-255b15446a5e","Type":"ContainerStarted","Data":"717a2a0bcad8941691399670f3f5d36723857feb016248f10f5d87d0262eaec1"}
Sep 09 21:01:32.058539 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:32.058270    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:01:32.083017 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:32.082964    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25" podStartSLOduration=0.88463433 podStartE2EDuration="5.082946969s" totalImagesPullingTime="4.198312639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:32.080630913 +0000 UTC m=+530.705104853" watchObservedRunningTime="2026-09-09 21:01:32.082946969 +0000 UTC m=+530.707420910"
Sep 09 21:01:43.065663 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:01:43.065586    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dzn25"
Sep 09 21:02:19.559952 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.559918    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"]
Sep 09 21:02:19.562422 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.562400    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:02:19.565290 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.565264    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 21:02:19.565497 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.565477    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bm2b9\""
Sep 09 21:02:19.572822 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.572799    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"]
Sep 09 21:02:19.590283 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.590252    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"]
Sep 09 21:02:19.650253 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.650207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9qz\" (UniqueName: \"kubernetes.io/projected/310ead78-ec2a-4d3d-a74d-12ebab9e711b-kube-api-access-tc9qz\") pod \"limitador-limitador-84994f7ffc-jw8rn\" (UID: \"310ead78-ec2a-4d3d-a74d-12ebab9e711b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:02:19.650438 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.650272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/310ead78-ec2a-4d3d-a74d-12ebab9e711b-config-file\") pod \"limitador-limitador-84994f7ffc-jw8rn\" (UID: \"310ead78-ec2a-4d3d-a74d-12ebab9e711b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:02:19.751259 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.751218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9qz\" (UniqueName: \"kubernetes.io/projected/310ead78-ec2a-4d3d-a74d-12ebab9e711b-kube-api-access-tc9qz\") pod \"limitador-limitador-84994f7ffc-jw8rn\" (UID: \"310ead78-ec2a-4d3d-a74d-12ebab9e711b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:02:19.751453 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.751271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/310ead78-ec2a-4d3d-a74d-12ebab9e711b-config-file\") pod \"limitador-limitador-84994f7ffc-jw8rn\" (UID: \"310ead78-ec2a-4d3d-a74d-12ebab9e711b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:02:19.751903 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.751877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/310ead78-ec2a-4d3d-a74d-12ebab9e711b-config-file\") pod \"limitador-limitador-84994f7ffc-jw8rn\" (UID: \"310ead78-ec2a-4d3d-a74d-12ebab9e711b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:02:19.760046 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.760019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9qz\" (UniqueName: \"kubernetes.io/projected/310ead78-ec2a-4d3d-a74d-12ebab9e711b-kube-api-access-tc9qz\") pod \"limitador-limitador-84994f7ffc-jw8rn\" (UID: \"310ead78-ec2a-4d3d-a74d-12ebab9e711b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:02:19.872786 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:19.872702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:02:20.150853 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:20.150780    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"]
Sep 09 21:02:20.154645 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:02:20.154606    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310ead78_ec2a_4d3d_a74d_12ebab9e711b.slice/crio-1305d1fb60274e0aff66278840e7abeec4052b6af109b72bfa6e5dcce11df947 WatchSource:0}: Error finding container 1305d1fb60274e0aff66278840e7abeec4052b6af109b72bfa6e5dcce11df947: Status 404 returned error can't find the container with id 1305d1fb60274e0aff66278840e7abeec4052b6af109b72bfa6e5dcce11df947
Sep 09 21:02:20.271091 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:20.271050    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn" event={"ID":"310ead78-ec2a-4d3d-a74d-12ebab9e711b","Type":"ContainerStarted","Data":"1305d1fb60274e0aff66278840e7abeec4052b6af109b72bfa6e5dcce11df947"}
Sep 09 21:02:21.276796 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:21.276697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn" event={"ID":"310ead78-ec2a-4d3d-a74d-12ebab9e711b","Type":"ContainerStarted","Data":"75805b0e05062f7dba8ce9ed209b056efe99b29dcec7bd222bb666123ad6572b"}
Sep 09 21:02:21.276796 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:21.276768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:02:21.298712 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:21.298664    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn" podStartSLOduration=1.4677574230000001 podStartE2EDuration="2.298646273s" totalImagesPullingTime="830.88885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:21.29737272 +0000 UTC m=+579.921846661" watchObservedRunningTime="2026-09-09 21:02:21.298646273 +0000 UTC m=+579.923120215"
Sep 09 21:02:32.282326 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:02:32.282295    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw8rn"
Sep 09 21:06:40.285700 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.285664    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"]
Sep 09 21:06:40.288971 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.288949    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.292041 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.292023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-txj6g\""
Sep 09 21:06:40.293585 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.293559    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 09 21:06:40.293585 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.293574    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 09 21:06:40.293758 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.293620    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 09 21:06:40.302359 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.302330    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"]
Sep 09 21:06:40.373668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.373633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-dshm\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.373838 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.373683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-home\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.373838 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.373717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.373838 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.373762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.373838 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.373790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq2nf\" (UniqueName: \"kubernetes.io/projected/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-kube-api-access-wq2nf\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.373838 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.373830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.474372 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.474330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.474574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.474390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq2nf\" (UniqueName: \"kubernetes.io/projected/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-kube-api-access-wq2nf\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.474574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.474427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.474574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.474468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-dshm\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.474574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.474504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-home\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.474574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.474529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.474928 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.474896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.475055 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.475019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-home\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.475055 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.474999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.477076 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.477056    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-dshm\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.477342 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.477317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.484312 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.484291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq2nf\" (UniqueName: \"kubernetes.io/projected/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-kube-api-access-wq2nf\") pod \"fc-smoke-test-kserve-7f4687897-987ff\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.599420 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.599392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:40.899034 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.899010    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"]
Sep 09 21:06:40.901684 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:06:40.901653    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf72265_0bb7_4b06_b4d1_0926ff3ca8eb.slice/crio-3652166b3fd88c2cfa2576b8d9ab3b7b290b183dc247ae3ef19f147deee179d5 WatchSource:0}: Error finding container 3652166b3fd88c2cfa2576b8d9ab3b7b290b183dc247ae3ef19f147deee179d5: Status 404 returned error can't find the container with id 3652166b3fd88c2cfa2576b8d9ab3b7b290b183dc247ae3ef19f147deee179d5
Sep 09 21:06:40.903636 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:40.903618    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:06:41.450291 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:41.450255    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff" event={"ID":"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb","Type":"ContainerStarted","Data":"3652166b3fd88c2cfa2576b8d9ab3b7b290b183dc247ae3ef19f147deee179d5"}
Sep 09 21:06:43.462963 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:43.462914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff" event={"ID":"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb","Type":"ContainerStarted","Data":"974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c"}
Sep 09 21:06:43.486594 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:43.486536    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff" podStartSLOduration=1.6141613540000002 podStartE2EDuration="3.486517468s" totalImagesPullingTime="1.872356114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:43.485885027 +0000 UTC m=+842.110358969" watchObservedRunningTime="2026-09-09 21:06:43.486517468 +0000 UTC m=+842.110991410"
Sep 09 21:06:43.599851 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:43.599817    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:44.957311 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:44.957270    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:06:44.957696 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:44.957358    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" podUID="d138945c-aa5e-4fb7-86c5-48e959a29350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:06:46.480134 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:46.480016    2583 generic.go:356] "Generic (PLEG): container finished" podID="d138945c-aa5e-4fb7-86c5-48e959a29350" containerID="69b936e399f7728f9187fe0dc7711d4694c510c443c7c0f1a9f552b4a5c6ff0b" exitCode=0
Sep 09 21:06:46.480134 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:46.480069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" event={"ID":"d138945c-aa5e-4fb7-86c5-48e959a29350","Type":"ContainerDied","Data":"69b936e399f7728f9187fe0dc7711d4694c510c443c7c0f1a9f552b4a5c6ff0b"}
Sep 09 21:06:46.480680 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:46.480592    2583 scope.go:117] "RemoveContainer" containerID="69b936e399f7728f9187fe0dc7711d4694c510c443c7c0f1a9f552b4a5c6ff0b"
Sep 09 21:06:46.481662 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:46.481631    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:06:46.481797 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:46.481780    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:06:46.481855 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:46.481821    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:06:47.487026 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:47.486993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" event={"ID":"d138945c-aa5e-4fb7-86c5-48e959a29350","Type":"ContainerStarted","Data":"a458760ba18b06e23a9920a814f999aab22b3e4ac1b413aab0ca46fb9f22a3e3"}
Sep 09 21:06:48.030893 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:48.030331    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:06:48.030893 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:48.030468    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 09 21:06:48.030893 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:48.030512    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" podUID="d138945c-aa5e-4fb7-86c5-48e959a29350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 09 21:06:49.032978 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:49.032943    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 09 21:06:49.033648 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:49.033558    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" podUID="d138945c-aa5e-4fb7-86c5-48e959a29350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 09 21:06:50.030534 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:50.030497    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 09 21:06:50.030820 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:50.030792    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" podUID="d138945c-aa5e-4fb7-86c5-48e959a29350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 09 21:06:50.599914 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:50.599881    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:50.612669 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:50.612636    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:51.029647 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:51.029549    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 09 21:06:51.029915 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:51.029880    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" podUID="d138945c-aa5e-4fb7-86c5-48e959a29350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 09 21:06:51.524416 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:51.524386    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:06:52.029601 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:52.029563    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 09 21:06:52.030235 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:52.030204    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k" podUID="d138945c-aa5e-4fb7-86c5-48e959a29350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 09 21:06:53.033338 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:53.033310    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:06:53.033769 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:53.033583    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:06:53.034369 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:06:53.034348    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k"
Sep 09 21:07:21.680598 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:21.680565    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"]
Sep 09 21:07:21.689990 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:07:21.689956    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:07:21.689990 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:07:21.689990 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:07:21.689990 ip-10-0-140-132 kubenswrapper[2583]:  > containerID="974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c" cmd=["/bin/sleep","15"]
Sep 09 21:07:21.690304 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:21.690019    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff" message="" err=<
Sep 09 21:07:21.690304 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:07:21.690304 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:07:21.690304 ip-10-0-140-132 kubenswrapper[2583]:  >
Sep 09 21:07:21.690304 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:07:21.690035    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:07:21.690304 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:07:21.690304 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:07:21.690304 ip-10-0-140-132 kubenswrapper[2583]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff" podUID="bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" containerName="main" containerID="cri-o://974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c"
Sep 09 21:07:21.690304 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:21.690079    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff" podUID="bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" containerName="main" containerID="cri-o://974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c" gracePeriod=60
Sep 09 21:07:22.030555 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.030529    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:07:22.137799 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.137751    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq2nf\" (UniqueName: \"kubernetes.io/projected/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-kube-api-access-wq2nf\") pod \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") "
Sep 09 21:07:22.137983 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.137855    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-dshm\") pod \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") "
Sep 09 21:07:22.137983 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.137917    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-model-cache\") pod \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") "
Sep 09 21:07:22.137983 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.137943    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-home\") pod \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") "
Sep 09 21:07:22.137983 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.137978    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tmp-dir\") pod \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") "
Sep 09 21:07:22.138248 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.138042    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tls-certs\") pod \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\" (UID: \"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb\") "
Sep 09 21:07:22.138504 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.138365    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-home" (OuterVolumeSpecName: "home") pod "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" (UID: "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:22.138643 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.138523    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" (UID: "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:22.138643 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.138601    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" (UID: "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:22.140775 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.140742    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-dshm" (OuterVolumeSpecName: "dshm") pod "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" (UID: "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:22.140877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.140852    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-kube-api-access-wq2nf" (OuterVolumeSpecName: "kube-api-access-wq2nf") pod "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" (UID: "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb"). InnerVolumeSpecName "kube-api-access-wq2nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:22.140932 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.140857    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" (UID: "bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:22.239746 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.239659    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:07:22.239746 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.239688    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:07:22.239746 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.239703    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:07:22.239746 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.239737    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:07:22.239746 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.239745    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:07:22.240012 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.239759    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq2nf\" (UniqueName: \"kubernetes.io/projected/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb-kube-api-access-wq2nf\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:07:22.657288 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.657251    2583 generic.go:356] "Generic (PLEG): container finished" podID="bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" containerID="974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c" exitCode=0
Sep 09 21:07:22.657468 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.657299    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"
Sep 09 21:07:22.657468 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.657333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff" event={"ID":"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb","Type":"ContainerDied","Data":"974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c"}
Sep 09 21:07:22.657468 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.657390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff" event={"ID":"bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb","Type":"ContainerDied","Data":"3652166b3fd88c2cfa2576b8d9ab3b7b290b183dc247ae3ef19f147deee179d5"}
Sep 09 21:07:22.657468 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.657412    2583 scope.go:117] "RemoveContainer" containerID="974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c"
Sep 09 21:07:22.667049 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.667030    2583 scope.go:117] "RemoveContainer" containerID="974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c"
Sep 09 21:07:22.667338 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:07:22.667319    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c\": container with ID starting with 974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c not found: ID does not exist" containerID="974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c"
Sep 09 21:07:22.667392 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.667351    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c"} err="failed to get container status \"974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c\": rpc error: code = NotFound desc = could not find container \"974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c\": container with ID starting with 974d7949bc8da292b344c89a63886cac71a6939e64ec1e25cbb5b8b524b4695c not found: ID does not exist"
Sep 09 21:07:22.682951 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.682924    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"]
Sep 09 21:07:22.684521 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:22.684497    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-987ff"]
Sep 09 21:07:23.865047 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:07:23.865008    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" path="/var/lib/kubelet/pods/bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb/volumes"
Sep 09 21:08:19.438287 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.438251    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"]
Sep 09 21:08:19.438754 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.438608    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" containerName="main"
Sep 09 21:08:19.438754 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.438623    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" containerName="main"
Sep 09 21:08:19.438754 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.438705    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdf72265-0bb7-4b06-b4d1-0926ff3ca8eb" containerName="main"
Sep 09 21:08:19.441735 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.441713    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.445597 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.445571    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 09 21:08:19.446755 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.446733    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 09 21:08:19.447273 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.447241    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 09 21:08:19.447375 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.447360    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-mcncw\""
Sep 09 21:08:19.459560 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.459531    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"]
Sep 09 21:08:19.545439 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.545400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.545439 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.545442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.545675 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.545530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzj6q\" (UniqueName: \"kubernetes.io/projected/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-kube-api-access-kzj6q\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.545675 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.545580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.545675 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.545638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-home\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.545675 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.545659    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.646529 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.646488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.646716 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.646553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-home\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.646716 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.646577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.646716 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.646605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.646716 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.646631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.646716 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.646687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzj6q\" (UniqueName: \"kubernetes.io/projected/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-kube-api-access-kzj6q\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.647037 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.647003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.647193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.647053    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-home\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.647193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.647072    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.649095 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.649072    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.649257 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.649242    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.656276 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.656249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzj6q\" (UniqueName: \"kubernetes.io/projected/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-kube-api-access-kzj6q\") pod \"fc-concurrency-test-kserve-b5d44467c-lvpm9\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:19.752655 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:19.752564    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:20.038605 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:20.038484    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"]
Sep 09 21:08:20.042185 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:08:20.042144    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab74f9b_7fe6_45f6_a7bc_186fd8ea979c.slice/crio-5fc007951c9c581e578f3dc2687d49d593185f69c88536f97b27dea8a6dbbc17 WatchSource:0}: Error finding container 5fc007951c9c581e578f3dc2687d49d593185f69c88536f97b27dea8a6dbbc17: Status 404 returned error can't find the container with id 5fc007951c9c581e578f3dc2687d49d593185f69c88536f97b27dea8a6dbbc17
Sep 09 21:08:20.924041 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:20.924003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9" event={"ID":"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c","Type":"ContainerStarted","Data":"ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf"}
Sep 09 21:08:20.924509 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:20.924061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9" event={"ID":"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c","Type":"ContainerStarted","Data":"5fc007951c9c581e578f3dc2687d49d593185f69c88536f97b27dea8a6dbbc17"}
Sep 09 21:08:21.752852 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:21.752817    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:29.752884 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:29.752845    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:29.765350 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:29.765323    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:29.787511 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:29.787459    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9" podStartSLOduration=10.787443565 podStartE2EDuration="10.787443565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:20.948327003 +0000 UTC m=+939.572800945" watchObservedRunningTime="2026-09-09 21:08:29.787443565 +0000 UTC m=+948.411917506"
Sep 09 21:08:29.975914 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:29.975886    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:57.711043 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:57.710997    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"]
Sep 09 21:08:57.720482 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:08:57.720441    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:08:57.720482 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:08:57.720482 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:08:57.720482 ip-10-0-140-132 kubenswrapper[2583]:  > containerID="ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf" cmd=["/bin/sleep","15"]
Sep 09 21:08:57.720715 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:57.720492    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9" message="" err=<
Sep 09 21:08:57.720715 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:08:57.720715 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:08:57.720715 ip-10-0-140-132 kubenswrapper[2583]:  >
Sep 09 21:08:57.720715 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:08:57.720505    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:08:57.720715 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:08:57.720715 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:08:57.720715 ip-10-0-140-132 kubenswrapper[2583]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9" podUID="1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" containerName="main" containerID="cri-o://ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf"
Sep 09 21:08:57.720715 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:57.720547    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9" podUID="1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" containerName="main" containerID="cri-o://ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf" gracePeriod=60
Sep 09 21:08:57.965061 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:57.964971    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9" podUID="1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": dial tcp 10.133.0.28:8000: connect: connection refused"
Sep 09 21:08:58.065930 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.065903    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:58.094232 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.094202    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" containerID="ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf" exitCode=0
Sep 09 21:08:58.094432 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.094243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9" event={"ID":"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c","Type":"ContainerDied","Data":"ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf"}
Sep 09 21:08:58.094432 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.094265    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"
Sep 09 21:08:58.094432 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.094288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9" event={"ID":"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c","Type":"ContainerDied","Data":"5fc007951c9c581e578f3dc2687d49d593185f69c88536f97b27dea8a6dbbc17"}
Sep 09 21:08:58.094432 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.094314    2583 scope.go:117] "RemoveContainer" containerID="ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf"
Sep 09 21:08:58.104232 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.104207    2583 scope.go:117] "RemoveContainer" containerID="ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf"
Sep 09 21:08:58.104529 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:08:58.104510    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf\": container with ID starting with ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf not found: ID does not exist" containerID="ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf"
Sep 09 21:08:58.104594 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.104544    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf"} err="failed to get container status \"ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf\": rpc error: code = NotFound desc = could not find container \"ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf\": container with ID starting with ef104c7465e1a1acb34e944a0428cfc1118f5f407d693b66b1372f3055a596cf not found: ID does not exist"
Sep 09 21:08:58.191442 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191413    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzj6q\" (UniqueName: \"kubernetes.io/projected/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-kube-api-access-kzj6q\") pod \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") "
Sep 09 21:08:58.191634 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191458    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tls-certs\") pod \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") "
Sep 09 21:08:58.191634 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191500    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-dshm\") pod \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") "
Sep 09 21:08:58.191634 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tmp-dir\") pod \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") "
Sep 09 21:08:58.191634 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-home\") pod \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") "
Sep 09 21:08:58.191840 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191674    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-model-cache\") pod \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\" (UID: \"1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c\") "
Sep 09 21:08:58.191908 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191850    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" (UID: "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:58.191986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191934    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-home" (OuterVolumeSpecName: "home") pod "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" (UID: "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:58.191986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191969    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:08:58.192136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.191993    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" (UID: "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:58.193973 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.193935    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-dshm" (OuterVolumeSpecName: "dshm") pod "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" (UID: "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:58.193973 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.193938    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" (UID: "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:58.193973 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.193966    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-kube-api-access-kzj6q" (OuterVolumeSpecName: "kube-api-access-kzj6q") pod "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" (UID: "1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c"). InnerVolumeSpecName "kube-api-access-kzj6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:58.293306 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.293273    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:08:58.293306 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.293303    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzj6q\" (UniqueName: \"kubernetes.io/projected/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-kube-api-access-kzj6q\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:08:58.293505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.293315    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:08:58.293505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.293327    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:08:58.293505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.293336    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:08:58.420631 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.420598    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"]
Sep 09 21:08:58.424056 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:58.424028    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lvpm9"]
Sep 09 21:08:59.865441 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:08:59.865403    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" path="/var/lib/kubelet/pods/1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c/volumes"
Sep 09 21:09:32.235187 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.235152    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 21:09:32.235658 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.235551    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" containerName="main"
Sep 09 21:09:32.235658 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.235564    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" containerName="main"
Sep 09 21:09:32.235658 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.235634    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab74f9b-7fe6-45f6-a7bc-186fd8ea979c" containerName="main"
Sep 09 21:09:32.238554 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.238526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.247220 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.247183    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-vvgkn\""
Sep 09 21:09:32.248623 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.248595    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 09 21:09:32.248773 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.248595    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 09 21:09:32.248773 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.248655    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 09 21:09:32.248773 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.248677    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-vf5nm\""
Sep 09 21:09:32.248773 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.248655    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 09 21:09:32.249681 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.249661    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:09:32.257432 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.257410    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 21:09:32.295246 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.295210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.295480 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.295298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.295480 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.295364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.295480 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.295416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.295655 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.295500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.295655 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.295558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44q9g\" (UniqueName: \"kubernetes.io/projected/dead927f-f6a8-4cc1-9a7d-878d93526339-kube-api-access-44q9g\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.295655 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.295644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dead927f-f6a8-4cc1-9a7d-878d93526339-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.295774 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.295687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396387 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396387 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396659 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396659 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44q9g\" (UniqueName: \"kubernetes.io/projected/dead927f-f6a8-4cc1-9a7d-878d93526339-kube-api-access-44q9g\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396659 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dead927f-f6a8-4cc1-9a7d-878d93526339-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396659 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396868 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396868 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396981 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396904    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.396981 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.396959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.397158 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.397131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.397158 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.397142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dead927f-f6a8-4cc1-9a7d-878d93526339-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.397312 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.397158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.398848 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.398829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.399042 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.399024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.408556 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.408533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44q9g\" (UniqueName: \"kubernetes.io/projected/dead927f-f6a8-4cc1-9a7d-878d93526339-kube-api-access-44q9g\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.549018 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.548991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:09:32.830129 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:32.830080    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 21:09:32.831965 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:09:32.831936    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddead927f_f6a8_4cc1_9a7d_878d93526339.slice/crio-c5611f9dcd104522a0e32e4302e705593ad4a09ddf40b88bb9d6fa0b0c19fee4 WatchSource:0}: Error finding container c5611f9dcd104522a0e32e4302e705593ad4a09ddf40b88bb9d6fa0b0c19fee4: Status 404 returned error can't find the container with id c5611f9dcd104522a0e32e4302e705593ad4a09ddf40b88bb9d6fa0b0c19fee4
Sep 09 21:09:33.273355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:33.273272    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"dead927f-f6a8-4cc1-9a7d-878d93526339","Type":"ContainerStarted","Data":"c5611f9dcd104522a0e32e4302e705593ad4a09ddf40b88bb9d6fa0b0c19fee4"}
Sep 09 21:09:37.303602 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:37.303561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"dead927f-f6a8-4cc1-9a7d-878d93526339","Type":"ContainerStarted","Data":"e725e908c94b1fb1eaf1a60cbedaae8e7a6f0a051a35ffbd490a7a7dffa0ecfa"}
Sep 09 21:09:39.314701 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:39.314665    2583 generic.go:356] "Generic (PLEG): container finished" podID="dead927f-f6a8-4cc1-9a7d-878d93526339" containerID="e725e908c94b1fb1eaf1a60cbedaae8e7a6f0a051a35ffbd490a7a7dffa0ecfa" exitCode=0
Sep 09 21:09:39.315076 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:09:39.314721    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"dead927f-f6a8-4cc1-9a7d-878d93526339","Type":"ContainerDied","Data":"e725e908c94b1fb1eaf1a60cbedaae8e7a6f0a051a35ffbd490a7a7dffa0ecfa"}
Sep 09 21:10:10.033508 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.032717    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"]
Sep 09 21:10:10.090506 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.090471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"]
Sep 09 21:10:10.090688 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.090590    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.094915 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.094863    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 09 21:10:10.095152 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.094924    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:10:10.095152 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.095005    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-8x8lh\""
Sep 09 21:10:10.095345 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.095310    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 09 21:10:10.162756 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.162723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.162937 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.162794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.162937 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.162843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.162937 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.162864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.162937 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.162900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btrgj\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-kube-api-access-btrgj\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.162937 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.162926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.163098 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.162964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.163098 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.162996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.163098 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.163013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.263923 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.263892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.263946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.263984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264336 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264336 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264336 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264336 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btrgj\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-kube-api-access-btrgj\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264545 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264378    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264545 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264441    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264672 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264729 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264672    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.264922 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.264899    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.266670 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.266651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.266817 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.266800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.273607 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.273573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.273705 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.273633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btrgj\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-kube-api-access-btrgj\") pod \"router-gateway-1-openshift-default-66bbccc5fb-hrwqn\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.403968 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.403927    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:10.903637 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.903576    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"]
Sep 09 21:10:10.908039 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:10:10.908005    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3afcf4_ad07_4476_9e5f_2ab4a59d97e6.slice/crio-1294b967f18b56371067847072332d648ed57925ab6f3f5bd66f7be33aa15388 WatchSource:0}: Error finding container 1294b967f18b56371067847072332d648ed57925ab6f3f5bd66f7be33aa15388: Status 404 returned error can't find the container with id 1294b967f18b56371067847072332d648ed57925ab6f3f5bd66f7be33aa15388
Sep 09 21:10:10.910787 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.910744    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:10:10.910910 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.910813    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:10:10.910910 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:10.910842    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:10:11.501773 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:11.501727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"dead927f-f6a8-4cc1-9a7d-878d93526339","Type":"ContainerStarted","Data":"88c1cafad3fe186412606915d09d240312af0b11a8a2c8df8815b2b1f20ac548"}
Sep 09 21:10:11.503392 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:11.503365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn" event={"ID":"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6","Type":"ContainerStarted","Data":"6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c"}
Sep 09 21:10:11.503515 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:11.503406    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn" event={"ID":"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6","Type":"ContainerStarted","Data":"1294b967f18b56371067847072332d648ed57925ab6f3f5bd66f7be33aa15388"}
Sep 09 21:10:11.559121 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:11.559056    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.296318322 podStartE2EDuration="39.559034297s" totalImagesPullingTime="35.121165191s" totalInitContainerRuntime="2.141550784s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:11.529620439 +0000 UTC m=+1050.154094382" watchObservedRunningTime="2026-09-09 21:10:11.559034297 +0000 UTC m=+1050.183508239"
Sep 09 21:10:11.560197 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:11.560165    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn" podStartSLOduration=1.560155465 podStartE2EDuration="1.560155465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:11.558652105 +0000 UTC m=+1050.183126049" watchObservedRunningTime="2026-09-09 21:10:11.560155465 +0000 UTC m=+1050.184629407"
Sep 09 21:10:12.404787 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:12.404747    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:12.409988 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:12.409962    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:12.507982 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:12.507947    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:12.509064 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:12.509041    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:52.998081 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:52.998026    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"]
Sep 09 21:10:52.998612 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:52.998389    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn" podUID="9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" containerName="istio-proxy" containerID="cri-o://6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c" gracePeriod=30
Sep 09 21:10:55.560592 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.560558    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:55.595466 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-data\") pod \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") "
Sep 09 21:10:55.595466 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595455    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-podinfo\") pod \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") "
Sep 09 21:10:55.595758 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595507    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-socket\") pod \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") "
Sep 09 21:10:55.595758 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595538    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btrgj\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-kube-api-access-btrgj\") pod \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") "
Sep 09 21:10:55.595758 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595571    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-certs\") pod \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") "
Sep 09 21:10:55.595758 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595601    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-envoy\") pod \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") "
Sep 09 21:10:55.595758 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595626    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-credential-socket\") pod \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") "
Sep 09 21:10:55.595758 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595630    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" (UID: "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:55.595758 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595696    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istiod-ca-cert\") pod \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") "
Sep 09 21:10:55.595758 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595745    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-token\") pod \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\" (UID: \"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6\") "
Sep 09 21:10:55.596294 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595796    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" (UID: "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:55.596294 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.595953    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" (UID: "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:55.596294 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.596053    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-credential-socket\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:10:55.596294 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.596083    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-data\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:10:55.596294 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.596098    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-socket\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:10:55.596592 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.596565    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" (UID: "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:55.598288 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.598258    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" (UID: "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:55.600590 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.600534    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" (UID: "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:10:55.600994 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.600967    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" (UID: "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:55.601090 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.601066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" (UID: "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:55.601369 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.601344    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-kube-api-access-btrgj" (OuterVolumeSpecName: "kube-api-access-btrgj") pod "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" (UID: "9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6"). InnerVolumeSpecName "kube-api-access-btrgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:55.696761 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.696719    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-podinfo\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:10:55.696944 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.696773    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btrgj\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-kube-api-access-btrgj\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:10:55.696944 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.696788    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-workload-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:10:55.696944 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.696799    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-envoy\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:10:55.696944 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.696810    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istiod-ca-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:10:55.696944 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.696819    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6-istio-token\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:10:55.721798 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.721765    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" containerID="6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c" exitCode=0
Sep 09 21:10:55.721979 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.721811    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"
Sep 09 21:10:55.721979 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.721857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn" event={"ID":"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6","Type":"ContainerDied","Data":"6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c"}
Sep 09 21:10:55.721979 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.721915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn" event={"ID":"9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6","Type":"ContainerDied","Data":"1294b967f18b56371067847072332d648ed57925ab6f3f5bd66f7be33aa15388"}
Sep 09 21:10:55.721979 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.721940    2583 scope.go:117] "RemoveContainer" containerID="6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c"
Sep 09 21:10:55.731998 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.731975    2583 scope.go:117] "RemoveContainer" containerID="6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c"
Sep 09 21:10:55.732350 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:10:55.732330    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c\": container with ID starting with 6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c not found: ID does not exist" containerID="6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c"
Sep 09 21:10:55.732410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.732363    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c"} err="failed to get container status \"6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c\": rpc error: code = NotFound desc = could not find container \"6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c\": container with ID starting with 6ecba797430a12ada39ed04beeabb4b15da13073f9fe29cb83d14ae89603497c not found: ID does not exist"
Sep 09 21:10:55.755124 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.755075    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"]
Sep 09 21:10:55.758417 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.758389    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-hrwqn"]
Sep 09 21:10:55.865525 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:10:55.865452    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" path="/var/lib/kubelet/pods/9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6/volumes"
Sep 09 21:11:35.859666 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:35.859619    2583 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-vf5nm\" not found"
Sep 09 21:11:35.979539 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:35.979485    2583 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:35.979741 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:35.979564    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs podName:dead927f-f6a8-4cc1-9a7d-878d93526339 nodeName:}" failed. No retries permitted until 2026-09-09 21:11:36.479547551 +0000 UTC m=+1135.104021470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:36.484979 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:36.484943    2583 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:36.485204 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:36.485008    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs podName:dead927f-f6a8-4cc1-9a7d-878d93526339 nodeName:}" failed. No retries permitted until 2026-09-09 21:11:37.484994224 +0000 UTC m=+1136.109468142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:37.495832 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:37.495791    2583 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:37.496320 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:37.495888    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs podName:dead927f-f6a8-4cc1-9a7d-878d93526339 nodeName:}" failed. No retries permitted until 2026-09-09 21:11:39.495863919 +0000 UTC m=+1138.120337857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:38.290340 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:38.290302    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 21:11:39.514065 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:39.514027    2583 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:39.514579 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:39.514093    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs podName:dead927f-f6a8-4cc1-9a7d-878d93526339 nodeName:}" failed. No retries permitted until 2026-09-09 21:11:43.514079434 +0000 UTC m=+1142.138553352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:43.554909 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:43.554854    2583 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:43.555526 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:43.554984    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs podName:dead927f-f6a8-4cc1-9a7d-878d93526339 nodeName:}" failed. No retries permitted until 2026-09-09 21:11:51.554959916 +0000 UTC m=+1150.179433846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:51.630692 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:51.630658    2583 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:51.631090 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:11:51.630727    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs podName:dead927f-f6a8-4cc1-9a7d-878d93526339 nodeName:}" failed. No retries permitted until 2026-09-09 21:12:07.630710756 +0000 UTC m=+1166.255184675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 21:11:53.313624 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:53.313563    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="dead927f-f6a8-4cc1-9a7d-878d93526339" containerName="main" containerID="cri-o://88c1cafad3fe186412606915d09d240312af0b11a8a2c8df8815b2b1f20ac548" gracePeriod=45
Sep 09 21:11:54.016594 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.016562    2583 generic.go:356] "Generic (PLEG): container finished" podID="dead927f-f6a8-4cc1-9a7d-878d93526339" containerID="88c1cafad3fe186412606915d09d240312af0b11a8a2c8df8815b2b1f20ac548" exitCode=0
Sep 09 21:11:54.016774 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.016641    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"dead927f-f6a8-4cc1-9a7d-878d93526339","Type":"ContainerDied","Data":"88c1cafad3fe186412606915d09d240312af0b11a8a2c8df8815b2b1f20ac548"}
Sep 09 21:11:54.054647 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.054625    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:11:54.155883 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.155790    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-home\") pod \"dead927f-f6a8-4cc1-9a7d-878d93526339\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") "
Sep 09 21:11:54.155883 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.155841    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-tmp-dir\") pod \"dead927f-f6a8-4cc1-9a7d-878d93526339\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") "
Sep 09 21:11:54.155883 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.155872    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-dshm\") pod \"dead927f-f6a8-4cc1-9a7d-878d93526339\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") "
Sep 09 21:11:54.156201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.155896    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dead927f-f6a8-4cc1-9a7d-878d93526339-cabundle-cert\") pod \"dead927f-f6a8-4cc1-9a7d-878d93526339\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") "
Sep 09 21:11:54.156201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.155969    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-kserve-provision-location\") pod \"dead927f-f6a8-4cc1-9a7d-878d93526339\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") "
Sep 09 21:11:54.156201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.155991    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-model-cache\") pod \"dead927f-f6a8-4cc1-9a7d-878d93526339\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") "
Sep 09 21:11:54.156201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.156171    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44q9g\" (UniqueName: \"kubernetes.io/projected/dead927f-f6a8-4cc1-9a7d-878d93526339-kube-api-access-44q9g\") pod \"dead927f-f6a8-4cc1-9a7d-878d93526339\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") "
Sep 09 21:11:54.156408 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.156220    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs\") pod \"dead927f-f6a8-4cc1-9a7d-878d93526339\" (UID: \"dead927f-f6a8-4cc1-9a7d-878d93526339\") "
Sep 09 21:11:54.156408 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.156309    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-model-cache" (OuterVolumeSpecName: "model-cache") pod "dead927f-f6a8-4cc1-9a7d-878d93526339" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:54.156511 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.156429    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dead927f-f6a8-4cc1-9a7d-878d93526339-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dead927f-f6a8-4cc1-9a7d-878d93526339" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:11:54.156596 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.156576    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:11:54.156654 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.156604    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dead927f-f6a8-4cc1-9a7d-878d93526339-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:11:54.156654 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.156611    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-home" (OuterVolumeSpecName: "home") pod "dead927f-f6a8-4cc1-9a7d-878d93526339" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:54.158769 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.158736    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dead927f-f6a8-4cc1-9a7d-878d93526339-kube-api-access-44q9g" (OuterVolumeSpecName: "kube-api-access-44q9g") pod "dead927f-f6a8-4cc1-9a7d-878d93526339" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339"). InnerVolumeSpecName "kube-api-access-44q9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:11:54.158902 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.158868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dead927f-f6a8-4cc1-9a7d-878d93526339" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:11:54.158902 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.158874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-dshm" (OuterVolumeSpecName: "dshm") pod "dead927f-f6a8-4cc1-9a7d-878d93526339" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:54.168396 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.168349    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dead927f-f6a8-4cc1-9a7d-878d93526339" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:54.176442 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.176413    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dead927f-f6a8-4cc1-9a7d-878d93526339" (UID: "dead927f-f6a8-4cc1-9a7d-878d93526339"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:54.257813 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.257765    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:11:54.257813 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.257805    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:11:54.257813 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.257815    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:11:54.257813 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.257823    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dead927f-f6a8-4cc1-9a7d-878d93526339-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:11:54.257813 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.257831    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44q9g\" (UniqueName: \"kubernetes.io/projected/dead927f-f6a8-4cc1-9a7d-878d93526339-kube-api-access-44q9g\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:11:54.258148 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:54.257839    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dead927f-f6a8-4cc1-9a7d-878d93526339-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:11:55.023791 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:55.023752    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"dead927f-f6a8-4cc1-9a7d-878d93526339","Type":"ContainerDied","Data":"c5611f9dcd104522a0e32e4302e705593ad4a09ddf40b88bb9d6fa0b0c19fee4"}
Sep 09 21:11:55.023791 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:55.023788    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 21:11:55.024338 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:55.023824    2583 scope.go:117] "RemoveContainer" containerID="88c1cafad3fe186412606915d09d240312af0b11a8a2c8df8815b2b1f20ac548"
Sep 09 21:11:55.032534 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:55.032517    2583 scope.go:117] "RemoveContainer" containerID="e725e908c94b1fb1eaf1a60cbedaae8e7a6f0a051a35ffbd490a7a7dffa0ecfa"
Sep 09 21:11:55.047562 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:55.047538    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 21:11:55.054706 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:55.054683    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 21:11:55.865171 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:11:55.865139    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dead927f-f6a8-4cc1-9a7d-878d93526339" path="/var/lib/kubelet/pods/dead927f-f6a8-4cc1-9a7d-878d93526339/volumes"
Sep 09 21:12:19.104792 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.104611    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"]
Sep 09 21:12:19.105668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.105380    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dead927f-f6a8-4cc1-9a7d-878d93526339" containerName="main"
Sep 09 21:12:19.105668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.105402    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dead927f-f6a8-4cc1-9a7d-878d93526339" containerName="main"
Sep 09 21:12:19.105668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.105437    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dead927f-f6a8-4cc1-9a7d-878d93526339" containerName="storage-initializer"
Sep 09 21:12:19.105668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.105447    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dead927f-f6a8-4cc1-9a7d-878d93526339" containerName="storage-initializer"
Sep 09 21:12:19.105668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.105456    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" containerName="istio-proxy"
Sep 09 21:12:19.105668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.105464    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" containerName="istio-proxy"
Sep 09 21:12:19.105668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.105569    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dead927f-f6a8-4cc1-9a7d-878d93526339" containerName="main"
Sep 09 21:12:19.105668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.105583    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b3afcf4-ad07-4476-9e5f-2ab4a59d97e6" containerName="istio-proxy"
Sep 09 21:12:19.108433 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.108413    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.112738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.112712    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-nrxw9\""
Sep 09 21:12:19.112738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.112726    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 09 21:12:19.112916 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.112764    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 09 21:12:19.112916 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.112782    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 09 21:12:19.124022 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.123996    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"]
Sep 09 21:12:19.180661 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.180614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.180870 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.180674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df76b2-56da-4b84-ad1f-983d716c923f-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.180870 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.180735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4b4t\" (UniqueName: \"kubernetes.io/projected/f8df76b2-56da-4b84-ad1f-983d716c923f-kube-api-access-m4b4t\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.180870 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.180811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.180870 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.180847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.181043 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.180906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.282291 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.282250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.282291 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.282293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.282541 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.282344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.282541 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.282369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.282541 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.282417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df76b2-56da-4b84-ad1f-983d716c923f-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.282541 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.282445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4b4t\" (UniqueName: \"kubernetes.io/projected/f8df76b2-56da-4b84-ad1f-983d716c923f-kube-api-access-m4b4t\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.282752 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.282714    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.282794 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.282762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.282832 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.282786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.284824 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.284804    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.285071 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.285054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df76b2-56da-4b84-ad1f-983d716c923f-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.292894 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.292871    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4b4t\" (UniqueName: \"kubernetes.io/projected/f8df76b2-56da-4b84-ad1f-983d716c923f-kube-api-access-m4b4t\") pod \"scheduler-inline-config-test-kserve-8655bc757-ttl7z\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.420449 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.420359    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:19.708676 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:12:19.708635    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8df76b2_56da_4b84_ad1f_983d716c923f.slice/crio-1650308dc4d0be03aa84ec7b932d3a60532e1386706871da83389a70e328ec4f WatchSource:0}: Error finding container 1650308dc4d0be03aa84ec7b932d3a60532e1386706871da83389a70e328ec4f: Status 404 returned error can't find the container with id 1650308dc4d0be03aa84ec7b932d3a60532e1386706871da83389a70e328ec4f
Sep 09 21:12:19.708828 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:19.708810    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"]
Sep 09 21:12:20.145009 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:20.144966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z" event={"ID":"f8df76b2-56da-4b84-ad1f-983d716c923f","Type":"ContainerStarted","Data":"d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55"}
Sep 09 21:12:20.145429 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:20.145035    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z" event={"ID":"f8df76b2-56da-4b84-ad1f-983d716c923f","Type":"ContainerStarted","Data":"1650308dc4d0be03aa84ec7b932d3a60532e1386706871da83389a70e328ec4f"}
Sep 09 21:12:20.174191 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:20.174137    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z" podStartSLOduration=1.174090144 podStartE2EDuration="1.174090144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:12:20.173955685 +0000 UTC m=+1178.798429626" watchObservedRunningTime="2026-09-09 21:12:20.174090144 +0000 UTC m=+1178.798564086"
Sep 09 21:12:20.420633 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:20.420546    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:29.421363 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:29.421307    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:29.433877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:29.433848    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:30.201132 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:30.201080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:53.207593 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:53.207559    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"]
Sep 09 21:12:53.216763 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:12:53.216724    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:12:53.216763 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:12:53.216763 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:12:53.216763 ip-10-0-140-132 kubenswrapper[2583]:  > containerID="d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55" cmd=["/bin/sleep","15"]
Sep 09 21:12:53.217079 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:53.216781    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z" message="" err=<
Sep 09 21:12:53.217079 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:12:53.217079 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:12:53.217079 ip-10-0-140-132 kubenswrapper[2583]:  >
Sep 09 21:12:53.217079 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:12:53.216794    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:12:53.217079 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:12:53.217079 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:12:53.217079 ip-10-0-140-132 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z" podUID="f8df76b2-56da-4b84-ad1f-983d716c923f" containerName="main" containerID="cri-o://d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55"
Sep 09 21:12:53.217079 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:53.216834    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z" podUID="f8df76b2-56da-4b84-ad1f-983d716c923f" containerName="main" containerID="cri-o://d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55" gracePeriod=60
Sep 09 21:12:53.973010 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:53.972982    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:54.094130 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.094018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-home\") pod \"f8df76b2-56da-4b84-ad1f-983d716c923f\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") "
Sep 09 21:12:54.094130 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.094051    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-dshm\") pod \"f8df76b2-56da-4b84-ad1f-983d716c923f\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") "
Sep 09 21:12:54.094130 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.094092    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4b4t\" (UniqueName: \"kubernetes.io/projected/f8df76b2-56da-4b84-ad1f-983d716c923f-kube-api-access-m4b4t\") pod \"f8df76b2-56da-4b84-ad1f-983d716c923f\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") "
Sep 09 21:12:54.094379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.094155    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-tmp-dir\") pod \"f8df76b2-56da-4b84-ad1f-983d716c923f\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") "
Sep 09 21:12:54.094379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.094194    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df76b2-56da-4b84-ad1f-983d716c923f-tls-certs\") pod \"f8df76b2-56da-4b84-ad1f-983d716c923f\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") "
Sep 09 21:12:54.094494 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.094373    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-home" (OuterVolumeSpecName: "home") pod "f8df76b2-56da-4b84-ad1f-983d716c923f" (UID: "f8df76b2-56da-4b84-ad1f-983d716c923f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:54.094494 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.094467    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8df76b2-56da-4b84-ad1f-983d716c923f" (UID: "f8df76b2-56da-4b84-ad1f-983d716c923f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:54.094595 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.094560    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-model-cache\") pod \"f8df76b2-56da-4b84-ad1f-983d716c923f\" (UID: \"f8df76b2-56da-4b84-ad1f-983d716c923f\") "
Sep 09 21:12:54.094845 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.094793    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8df76b2-56da-4b84-ad1f-983d716c923f" (UID: "f8df76b2-56da-4b84-ad1f-983d716c923f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:54.095089 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.095047    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.095157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.095096    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.095157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.095139    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.096535 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.096511    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-dshm" (OuterVolumeSpecName: "dshm") pod "f8df76b2-56da-4b84-ad1f-983d716c923f" (UID: "f8df76b2-56da-4b84-ad1f-983d716c923f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:54.096630 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.096612    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8df76b2-56da-4b84-ad1f-983d716c923f-kube-api-access-m4b4t" (OuterVolumeSpecName: "kube-api-access-m4b4t") pod "f8df76b2-56da-4b84-ad1f-983d716c923f" (UID: "f8df76b2-56da-4b84-ad1f-983d716c923f"). InnerVolumeSpecName "kube-api-access-m4b4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:54.096630 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.096618    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8df76b2-56da-4b84-ad1f-983d716c923f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8df76b2-56da-4b84-ad1f-983d716c923f" (UID: "f8df76b2-56da-4b84-ad1f-983d716c923f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:54.196430 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.196391    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df76b2-56da-4b84-ad1f-983d716c923f-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.196430 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.196423    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4b4t\" (UniqueName: \"kubernetes.io/projected/f8df76b2-56da-4b84-ad1f-983d716c923f-kube-api-access-m4b4t\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.196430 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.196434    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df76b2-56da-4b84-ad1f-983d716c923f-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.315677 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.315640    2583 generic.go:356] "Generic (PLEG): container finished" podID="f8df76b2-56da-4b84-ad1f-983d716c923f" containerID="d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55" exitCode=0
Sep 09 21:12:54.316074 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.315689    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z" event={"ID":"f8df76b2-56da-4b84-ad1f-983d716c923f","Type":"ContainerDied","Data":"d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55"}
Sep 09 21:12:54.316074 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.315704    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"
Sep 09 21:12:54.316074 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.315731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z" event={"ID":"f8df76b2-56da-4b84-ad1f-983d716c923f","Type":"ContainerDied","Data":"1650308dc4d0be03aa84ec7b932d3a60532e1386706871da83389a70e328ec4f"}
Sep 09 21:12:54.316074 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.315753    2583 scope.go:117] "RemoveContainer" containerID="d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55"
Sep 09 21:12:54.325418 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.325400    2583 scope.go:117] "RemoveContainer" containerID="d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55"
Sep 09 21:12:54.325691 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:12:54.325674    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55\": container with ID starting with d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55 not found: ID does not exist" containerID="d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55"
Sep 09 21:12:54.325740 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.325701    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55"} err="failed to get container status \"d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55\": rpc error: code = NotFound desc = could not find container \"d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55\": container with ID starting with d2119ffdda3991c7b1c9dd17dd1da5cc23fc701b2531aee9db2a47c917b5ed55 not found: ID does not exist"
Sep 09 21:12:54.342968 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.342943    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"]
Sep 09 21:12:54.346660 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:54.346598    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-ttl7z"]
Sep 09 21:12:55.865438 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:12:55.865401    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8df76b2-56da-4b84-ad1f-983d716c923f" path="/var/lib/kubelet/pods/f8df76b2-56da-4b84-ad1f-983d716c923f/volumes"
Sep 09 21:14:11.105026 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.104993    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"]
Sep 09 21:14:11.105410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.105346    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8df76b2-56da-4b84-ad1f-983d716c923f" containerName="main"
Sep 09 21:14:11.105410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.105358    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8df76b2-56da-4b84-ad1f-983d716c923f" containerName="main"
Sep 09 21:14:11.105481 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.105424    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8df76b2-56da-4b84-ad1f-983d716c923f" containerName="main"
Sep 09 21:14:11.108313 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.108295    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.111084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.111061    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 09 21:14:11.111084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.111062    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 09 21:14:11.111299 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.111134    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-lpn6n\""
Sep 09 21:14:11.111299 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.111141    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 09 21:14:11.111299 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.111241    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:14:11.112310 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.112295    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 09 21:14:11.130430 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.130395    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"]
Sep 09 21:14:11.173257 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.173226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9k4\" (UniqueName: \"kubernetes.io/projected/5641f3fe-7f25-496f-a06a-60aea5560fe9-kube-api-access-kv9k4\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.173411 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.173271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-home\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.173411 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.173301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5641f3fe-7f25-496f-a06a-60aea5560fe9-tls-certs\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.173411 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.173367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-dshm\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.173411 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.173400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-model-cache\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.173544 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.173426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5641f3fe-7f25-496f-a06a-60aea5560fe9-cabundle-cert\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.173544 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.173495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-tmp-dir\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.173544 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.173519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.274636 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.274599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5641f3fe-7f25-496f-a06a-60aea5560fe9-tls-certs\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.274636 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.274650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-dshm\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.274848 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.274776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-model-cache\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.274848 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.274811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5641f3fe-7f25-496f-a06a-60aea5560fe9-cabundle-cert\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.274848 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.274843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-tmp-dir\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.275002 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.274866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.275002 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.274947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9k4\" (UniqueName: \"kubernetes.io/projected/5641f3fe-7f25-496f-a06a-60aea5560fe9-kube-api-access-kv9k4\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.275222 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.275195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-model-cache\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.275349 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.275295    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-tmp-dir\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.275349 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.275321    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.275463 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.275396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-home\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.275604 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.275582    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5641f3fe-7f25-496f-a06a-60aea5560fe9-cabundle-cert\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.275645 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.275610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-home\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.277179 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.277159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-dshm\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.277368 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.277351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5641f3fe-7f25-496f-a06a-60aea5560fe9-tls-certs\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.284939 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.284913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9k4\" (UniqueName: \"kubernetes.io/projected/5641f3fe-7f25-496f-a06a-60aea5560fe9-kube-api-access-kv9k4\") pod \"auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.419713 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.419620    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:11.702449 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:11.702266    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"]
Sep 09 21:14:11.705168 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:14:11.705142    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5641f3fe_7f25_496f_a06a_60aea5560fe9.slice/crio-788495b519209e09bb6dbf6832ee82977269aef1c0a58e61688f410cf865e534 WatchSource:0}: Error finding container 788495b519209e09bb6dbf6832ee82977269aef1c0a58e61688f410cf865e534: Status 404 returned error can't find the container with id 788495b519209e09bb6dbf6832ee82977269aef1c0a58e61688f410cf865e534
Sep 09 21:14:12.681425 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:12.681382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" event={"ID":"5641f3fe-7f25-496f-a06a-60aea5560fe9","Type":"ContainerStarted","Data":"643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429"}
Sep 09 21:14:12.681925 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:12.681446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" event={"ID":"5641f3fe-7f25-496f-a06a-60aea5560fe9","Type":"ContainerStarted","Data":"788495b519209e09bb6dbf6832ee82977269aef1c0a58e61688f410cf865e534"}
Sep 09 21:14:14.693596 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:14.693547    2583 generic.go:356] "Generic (PLEG): container finished" podID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerID="643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429" exitCode=0
Sep 09 21:14:14.693975 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:14.693618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" event={"ID":"5641f3fe-7f25-496f-a06a-60aea5560fe9","Type":"ContainerDied","Data":"643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429"}
Sep 09 21:14:15.699845 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:15.699808    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" event={"ID":"5641f3fe-7f25-496f-a06a-60aea5560fe9","Type":"ContainerStarted","Data":"6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67"}
Sep 09 21:14:15.729186 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:15.729099    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podStartSLOduration=2.504600546 podStartE2EDuration="4.729078193s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.224477647s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:15.727553885 +0000 UTC m=+1294.352027825" watchObservedRunningTime="2026-09-09 21:14:15.729078193 +0000 UTC m=+1294.353552137"
Sep 09 21:14:16.419956 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:16.419893    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:21.420699 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:21.420656    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:14:21.422218 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:21.422187    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:14:31.420639 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:31.420594    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:14:41.420303 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:41.420262    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:14:51.420842 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:14:51.420790    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:15:01.420575 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:15:01.420526    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:15:11.420651 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:15:11.420564    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:15:21.420126 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:15:21.420058    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:15:31.420186 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:15:31.420129    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:15:41.420497 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:15:41.420453    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:15:51.420408 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:15:51.420368    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:16:01.420726 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:01.420679    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:16:11.430240 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:11.430206    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:16:11.437730 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:11.437702    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:16:13.193072 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:13.193038    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"]
Sep 09 21:16:28.297967 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:28.297914    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" containerID="cri-o://6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67" gracePeriod=45
Sep 09 21:16:28.430954 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:28.430912    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 09 21:16:29.049733 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.049706    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:16:29.136710 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.136622    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5641f3fe-7f25-496f-a06a-60aea5560fe9-cabundle-cert\") pod \"5641f3fe-7f25-496f-a06a-60aea5560fe9\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") "
Sep 09 21:16:29.136710 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.136692    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv9k4\" (UniqueName: \"kubernetes.io/projected/5641f3fe-7f25-496f-a06a-60aea5560fe9-kube-api-access-kv9k4\") pod \"5641f3fe-7f25-496f-a06a-60aea5560fe9\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") "
Sep 09 21:16:29.136931 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.136726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-model-cache\") pod \"5641f3fe-7f25-496f-a06a-60aea5560fe9\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") "
Sep 09 21:16:29.136931 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.136746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-tmp-dir\") pod \"5641f3fe-7f25-496f-a06a-60aea5560fe9\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") "
Sep 09 21:16:29.136931 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.136820    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-home\") pod \"5641f3fe-7f25-496f-a06a-60aea5560fe9\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") "
Sep 09 21:16:29.136931 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.136853    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-kserve-provision-location\") pod \"5641f3fe-7f25-496f-a06a-60aea5560fe9\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") "
Sep 09 21:16:29.136931 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.136881    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-dshm\") pod \"5641f3fe-7f25-496f-a06a-60aea5560fe9\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") "
Sep 09 21:16:29.137229 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.136943    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5641f3fe-7f25-496f-a06a-60aea5560fe9-tls-certs\") pod \"5641f3fe-7f25-496f-a06a-60aea5560fe9\" (UID: \"5641f3fe-7f25-496f-a06a-60aea5560fe9\") "
Sep 09 21:16:29.137229 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.136963    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5641f3fe-7f25-496f-a06a-60aea5560fe9" (UID: "5641f3fe-7f25-496f-a06a-60aea5560fe9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:29.137229 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.137008    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5641f3fe-7f25-496f-a06a-60aea5560fe9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5641f3fe-7f25-496f-a06a-60aea5560fe9" (UID: "5641f3fe-7f25-496f-a06a-60aea5560fe9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:29.137378 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.137298    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5641f3fe-7f25-496f-a06a-60aea5560fe9-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.137378 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.137317    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.137653 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.137502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-home" (OuterVolumeSpecName: "home") pod "5641f3fe-7f25-496f-a06a-60aea5560fe9" (UID: "5641f3fe-7f25-496f-a06a-60aea5560fe9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:29.139460 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.139430    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5641f3fe-7f25-496f-a06a-60aea5560fe9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5641f3fe-7f25-496f-a06a-60aea5560fe9" (UID: "5641f3fe-7f25-496f-a06a-60aea5560fe9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:29.139719 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.139694    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-dshm" (OuterVolumeSpecName: "dshm") pod "5641f3fe-7f25-496f-a06a-60aea5560fe9" (UID: "5641f3fe-7f25-496f-a06a-60aea5560fe9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:29.139793 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.139769    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5641f3fe-7f25-496f-a06a-60aea5560fe9-kube-api-access-kv9k4" (OuterVolumeSpecName: "kube-api-access-kv9k4") pod "5641f3fe-7f25-496f-a06a-60aea5560fe9" (UID: "5641f3fe-7f25-496f-a06a-60aea5560fe9"). InnerVolumeSpecName "kube-api-access-kv9k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:29.155495 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.155447    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5641f3fe-7f25-496f-a06a-60aea5560fe9" (UID: "5641f3fe-7f25-496f-a06a-60aea5560fe9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:29.160441 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.160418    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5641f3fe-7f25-496f-a06a-60aea5560fe9" (UID: "5641f3fe-7f25-496f-a06a-60aea5560fe9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:29.238197 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.238159    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv9k4\" (UniqueName: \"kubernetes.io/projected/5641f3fe-7f25-496f-a06a-60aea5560fe9-kube-api-access-kv9k4\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.238197 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.238192    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.238197 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.238205    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.238430 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.238214    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.238430 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.238221    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5641f3fe-7f25-496f-a06a-60aea5560fe9-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.238430 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.238229    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5641f3fe-7f25-496f-a06a-60aea5560fe9-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.352355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.352323    2583 generic.go:356] "Generic (PLEG): container finished" podID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerID="6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67" exitCode=0
Sep 09 21:16:29.352739 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.352381    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"
Sep 09 21:16:29.352739 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.352385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" event={"ID":"5641f3fe-7f25-496f-a06a-60aea5560fe9","Type":"ContainerDied","Data":"6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67"}
Sep 09 21:16:29.352739 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.352521    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb" event={"ID":"5641f3fe-7f25-496f-a06a-60aea5560fe9","Type":"ContainerDied","Data":"788495b519209e09bb6dbf6832ee82977269aef1c0a58e61688f410cf865e534"}
Sep 09 21:16:29.352739 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.352542    2583 scope.go:117] "RemoveContainer" containerID="6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67"
Sep 09 21:16:29.361901 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.361885    2583 scope.go:117] "RemoveContainer" containerID="643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429"
Sep 09 21:16:29.369611 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.369594    2583 scope.go:117] "RemoveContainer" containerID="6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67"
Sep 09 21:16:29.369868 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:16:29.369847    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67\": container with ID starting with 6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67 not found: ID does not exist" containerID="6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67"
Sep 09 21:16:29.369930 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.369875    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67"} err="failed to get container status \"6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67\": rpc error: code = NotFound desc = could not find container \"6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67\": container with ID starting with 6d1fe5ba6419a534507d1ad232f03fe6d46aa91f22b32ad20687535e4d474e67 not found: ID does not exist"
Sep 09 21:16:29.369930 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.369892    2583 scope.go:117] "RemoveContainer" containerID="643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429"
Sep 09 21:16:29.370227 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:16:29.370206    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429\": container with ID starting with 643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429 not found: ID does not exist" containerID="643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429"
Sep 09 21:16:29.370278 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.370235    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429"} err="failed to get container status \"643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429\": rpc error: code = NotFound desc = could not find container \"643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429\": container with ID starting with 643eb185e67210cfe82c42f8756b7d2df59a8b931d86699796eff21fb0525429 not found: ID does not exist"
Sep 09 21:16:29.376746 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.376722    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"]
Sep 09 21:16:29.380377 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.380357    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7bd5cd56f8-x4qlb"]
Sep 09 21:16:29.866384 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:16:29.866351    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" path="/var/lib/kubelet/pods/5641f3fe-7f25-496f-a06a-60aea5560fe9/volumes"
Sep 09 21:17:11.083713 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.083682    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"]
Sep 09 21:17:11.084193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.084028    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="storage-initializer"
Sep 09 21:17:11.084193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.084041    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="storage-initializer"
Sep 09 21:17:11.084193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.084051    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main"
Sep 09 21:17:11.084193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.084057    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main"
Sep 09 21:17:11.084193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.084153    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5641f3fe-7f25-496f-a06a-60aea5560fe9" containerName="main"
Sep 09 21:17:11.088903 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.088875    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.094255 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.094227    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:17:11.094719 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.094681    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 09 21:17:11.094857 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.094728    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 09 21:17:11.094857 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.094746    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 09 21:17:11.094857 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.094691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-6g76z\""
Sep 09 21:17:11.095029 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.094970    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 09 21:17:11.102467 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.102438    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"]
Sep 09 21:17:11.203351 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.203318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4crt\" (UniqueName: \"kubernetes.io/projected/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kube-api-access-t4crt\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.203351 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.203354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-dshm\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.203596 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.203424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-model-cache\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.203596 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.203446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tmp-dir\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.203596 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.203481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kserve-provision-location\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.203596 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.203530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tls-certs\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.203596 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.203547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-cabundle-cert\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.203778 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.203615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-home\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.304994 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.304944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-model-cache\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.304994 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.304994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tmp-dir\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305300 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kserve-provision-location\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305300 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tls-certs\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305300 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-cabundle-cert\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305300 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-home\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305300 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4crt\" (UniqueName: \"kubernetes.io/projected/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kube-api-access-t4crt\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305300 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-dshm\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-model-cache\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tmp-dir\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kserve-provision-location\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305800 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-cabundle-cert\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.305873 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.305818    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-home\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.307727 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.307708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-dshm\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.307895 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.307879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tls-certs\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.315223 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.315199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4crt\" (UniqueName: \"kubernetes.io/projected/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kube-api-access-t4crt\") pod \"auth-disabled-test-kserve-65b7dcb96f-28kdl\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.401173 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.401043    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:11.685259 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:11.685071    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"]
Sep 09 21:17:11.688034 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:17:11.688004    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7833fa65_9b4b_4e27_bb73_d46d36e2f63a.slice/crio-f23ea8e224911c10149960bb16529baba32055ff97f517e03f3f176263b8eed5 WatchSource:0}: Error finding container f23ea8e224911c10149960bb16529baba32055ff97f517e03f3f176263b8eed5: Status 404 returned error can't find the container with id f23ea8e224911c10149960bb16529baba32055ff97f517e03f3f176263b8eed5
Sep 09 21:17:12.551597 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:12.551553    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" event={"ID":"7833fa65-9b4b-4e27-bb73-d46d36e2f63a","Type":"ContainerStarted","Data":"13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f"}
Sep 09 21:17:12.552079 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:12.551621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" event={"ID":"7833fa65-9b4b-4e27-bb73-d46d36e2f63a","Type":"ContainerStarted","Data":"f23ea8e224911c10149960bb16529baba32055ff97f517e03f3f176263b8eed5"}
Sep 09 21:17:14.564190 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:14.564151    2583 generic.go:356] "Generic (PLEG): container finished" podID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerID="13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f" exitCode=0
Sep 09 21:17:14.564571 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:14.564232    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" event={"ID":"7833fa65-9b4b-4e27-bb73-d46d36e2f63a","Type":"ContainerDied","Data":"13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f"}
Sep 09 21:17:15.571433 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:15.571396    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" event={"ID":"7833fa65-9b4b-4e27-bb73-d46d36e2f63a","Type":"ContainerStarted","Data":"1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51"}
Sep 09 21:17:15.598764 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:15.598690    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podStartSLOduration=2.33851619 podStartE2EDuration="4.598673119s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.260156929s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:17:15.596388729 +0000 UTC m=+1474.220862670" watchObservedRunningTime="2026-09-09 21:17:15.598673119 +0000 UTC m=+1474.223147060"
Sep 09 21:17:16.401848 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:16.401794    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:21.401209 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:21.401169    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:17:21.403052 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:21.403023    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:17:31.401736 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:31.401689    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:17:41.402388 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:41.402335    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:17:51.401655 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:17:51.401611    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:18:01.401645 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:18:01.401606    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:18:11.402017 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:18:11.401922    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:18:21.402373 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:18:21.402317    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:18:31.401873 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:18:31.401827    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:18:41.401469 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:18:41.401415    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:18:51.401641 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:18:51.401596    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:19:01.402070 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:01.402012    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 09 21:19:11.411518 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:11.411483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:19:11.419478 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:11.419447    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:19:16.659680 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:16.659645    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"]
Sep 09 21:19:31.683823 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:31.683754    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" containerID="cri-o://1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51" gracePeriod=45
Sep 09 21:19:32.435647 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.435623    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:19:32.475813 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.475739    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-model-cache\") pod \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") "
Sep 09 21:19:32.475813 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.475780    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kserve-provision-location\") pod \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") "
Sep 09 21:19:32.475813 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.475801    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tls-certs\") pod \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") "
Sep 09 21:19:32.476084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.475831    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-cabundle-cert\") pod \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") "
Sep 09 21:19:32.476084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.475882    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-dshm\") pod \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") "
Sep 09 21:19:32.476084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.475941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4crt\" (UniqueName: \"kubernetes.io/projected/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kube-api-access-t4crt\") pod \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") "
Sep 09 21:19:32.476084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.475977    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tmp-dir\") pod \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") "
Sep 09 21:19:32.476084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.476035    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-home\") pod \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\" (UID: \"7833fa65-9b4b-4e27-bb73-d46d36e2f63a\") "
Sep 09 21:19:32.476367 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.476092    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7833fa65-9b4b-4e27-bb73-d46d36e2f63a" (UID: "7833fa65-9b4b-4e27-bb73-d46d36e2f63a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:32.476367 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.476358    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:19:32.476477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.476357    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7833fa65-9b4b-4e27-bb73-d46d36e2f63a" (UID: "7833fa65-9b4b-4e27-bb73-d46d36e2f63a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:19:32.477184 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.477158    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-home" (OuterVolumeSpecName: "home") pod "7833fa65-9b4b-4e27-bb73-d46d36e2f63a" (UID: "7833fa65-9b4b-4e27-bb73-d46d36e2f63a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:32.478468 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.478445    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7833fa65-9b4b-4e27-bb73-d46d36e2f63a" (UID: "7833fa65-9b4b-4e27-bb73-d46d36e2f63a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:32.479086 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.479062    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-dshm" (OuterVolumeSpecName: "dshm") pod "7833fa65-9b4b-4e27-bb73-d46d36e2f63a" (UID: "7833fa65-9b4b-4e27-bb73-d46d36e2f63a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:32.479246 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.479082    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kube-api-access-t4crt" (OuterVolumeSpecName: "kube-api-access-t4crt") pod "7833fa65-9b4b-4e27-bb73-d46d36e2f63a" (UID: "7833fa65-9b4b-4e27-bb73-d46d36e2f63a"). InnerVolumeSpecName "kube-api-access-t4crt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:32.495967 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.495925    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7833fa65-9b4b-4e27-bb73-d46d36e2f63a" (UID: "7833fa65-9b4b-4e27-bb73-d46d36e2f63a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:32.502098 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.502069    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7833fa65-9b4b-4e27-bb73-d46d36e2f63a" (UID: "7833fa65-9b4b-4e27-bb73-d46d36e2f63a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:32.577389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.577317    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:19:32.577389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.577347    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:19:32.577389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.577359    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:19:32.577389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.577367    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:19:32.577389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.577375    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4crt\" (UniqueName: \"kubernetes.io/projected/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-kube-api-access-t4crt\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:19:32.577389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.577383    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:19:32.577389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:32.577393    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7833fa65-9b4b-4e27-bb73-d46d36e2f63a-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:19:33.240760 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.240727    2583 generic.go:356] "Generic (PLEG): container finished" podID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerID="1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51" exitCode=0
Sep 09 21:19:33.241158 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.240775    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"
Sep 09 21:19:33.241158 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.240805    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" event={"ID":"7833fa65-9b4b-4e27-bb73-d46d36e2f63a","Type":"ContainerDied","Data":"1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51"}
Sep 09 21:19:33.241158 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.240870    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" event={"ID":"7833fa65-9b4b-4e27-bb73-d46d36e2f63a","Type":"ContainerDied","Data":"f23ea8e224911c10149960bb16529baba32055ff97f517e03f3f176263b8eed5"}
Sep 09 21:19:33.241158 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.240893    2583 scope.go:117] "RemoveContainer" containerID="1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51"
Sep 09 21:19:33.250411 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.250390    2583 scope.go:117] "RemoveContainer" containerID="13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f"
Sep 09 21:19:33.258057 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.258037    2583 scope.go:117] "RemoveContainer" containerID="1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51"
Sep 09 21:19:33.258338 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:19:33.258320    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51\": container with ID starting with 1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51 not found: ID does not exist" containerID="1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51"
Sep 09 21:19:33.258390 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.258349    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51"} err="failed to get container status \"1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51\": rpc error: code = NotFound desc = could not find container \"1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51\": container with ID starting with 1186094b3f4671fe00310d22b804d058613606917d2a4237db9b5e5631c3cd51 not found: ID does not exist"
Sep 09 21:19:33.258390 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.258366    2583 scope.go:117] "RemoveContainer" containerID="13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f"
Sep 09 21:19:33.258607 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:19:33.258593    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f\": container with ID starting with 13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f not found: ID does not exist" containerID="13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f"
Sep 09 21:19:33.258644 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.258610    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f"} err="failed to get container status \"13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f\": rpc error: code = NotFound desc = could not find container \"13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f\": container with ID starting with 13b975c985a8f6a61d26a50df1d502b64d6e5bdcc420e1dd5a81e68cd5aea98f not found: ID does not exist"
Sep 09 21:19:33.270821 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.270790    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"]
Sep 09 21:19:33.273152 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.273126    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl"]
Sep 09 21:19:33.412480 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.412429    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65b7dcb96f-28kdl" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:19:33.865626 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:19:33.865590    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" path="/var/lib/kubelet/pods/7833fa65-9b4b-4e27-bb73-d46d36e2f63a/volumes"
Sep 09 21:20:08.945496 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.945463    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"]
Sep 09 21:20:08.946044 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.945984    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main"
Sep 09 21:20:08.946044 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.946004    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main"
Sep 09 21:20:08.946044 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.946036    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="storage-initializer"
Sep 09 21:20:08.946044 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.946044    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="storage-initializer"
Sep 09 21:20:08.946315 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.946148    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7833fa65-9b4b-4e27-bb73-d46d36e2f63a" containerName="main"
Sep 09 21:20:08.949451 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.949429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:08.955707 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.955674    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-6bqsz\""
Sep 09 21:20:08.956777 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.956758    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 09 21:20:08.956901 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.956882    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 09 21:20:08.956948 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.956919    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 09 21:20:08.957010 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.956989    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-pfchr\""
Sep 09 21:20:08.972052 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:08.972021    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"]
Sep 09 21:20:09.097362 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.097325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.097549 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.097383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.097549 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.097420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72t9z\" (UniqueName: \"kubernetes.io/projected/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-kube-api-access-72t9z\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.097649 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.097551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.097649 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.097601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.097715 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.097662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.198472 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.198382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.198472 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.198433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72t9z\" (UniqueName: \"kubernetes.io/projected/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-kube-api-access-72t9z\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.198707 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.198498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.198707 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.198535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.198707 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.198591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.198707 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.198647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.198921 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.198835    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.199012 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.198991    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.199156 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.199134    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.201522 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.201496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.201646 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.201610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.215935 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.215904    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72t9z\" (UniqueName: \"kubernetes.io/projected/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-kube-api-access-72t9z\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.258941 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.258903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:09.550850 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.550820    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"]
Sep 09 21:20:09.553075 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:20:09.553036    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b01a2f4_e42f_4cdc_b56c_b0ac27dc8fa4.slice/crio-cd74385f84e29d9bd80c75fd67309a20a89326ebc983751a794c1b03665eb99c WatchSource:0}: Error finding container cd74385f84e29d9bd80c75fd67309a20a89326ebc983751a794c1b03665eb99c: Status 404 returned error can't find the container with id cd74385f84e29d9bd80c75fd67309a20a89326ebc983751a794c1b03665eb99c
Sep 09 21:20:09.555162 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:09.555139    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:20:10.446391 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:10.446343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" event={"ID":"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4","Type":"ContainerStarted","Data":"cd74385f84e29d9bd80c75fd67309a20a89326ebc983751a794c1b03665eb99c"}
Sep 09 21:20:13.464122 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:13.464066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" event={"ID":"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4","Type":"ContainerStarted","Data":"22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8"}
Sep 09 21:20:13.464632 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:13.464236    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:14.470215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:14.470179    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" event={"ID":"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4","Type":"ContainerStarted","Data":"c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e"}
Sep 09 21:20:14.493829 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:14.493764    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" podStartSLOduration=3.501852295 podStartE2EDuration="6.493739289s" totalImagesPullingTime="2.991886994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:14.492188547 +0000 UTC m=+1653.116662488" watchObservedRunningTime="2026-09-09 21:20:14.493739289 +0000 UTC m=+1653.118213230"
Sep 09 21:20:15.259191 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:15.259150    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:19.259408 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:19.259350    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:19.271745 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:19.271719    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:19.505348 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:19.505322    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:20:30.510857 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:20:30.510825    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:21:59.638193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:21:59.638137    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"]
Sep 09 21:21:59.648025 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:21:59.647990    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:21:59.648025 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:21:59.648025 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:21:59.648025 ip-10-0-140-132 kubenswrapper[2583]:  > containerID="c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e" cmd=["/bin/sleep","15"]
Sep 09 21:21:59.648282 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:21:59.648045    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" message="" err=<
Sep 09 21:21:59.648282 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:21:59.648282 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:21:59.648282 ip-10-0-140-132 kubenswrapper[2583]:  >
Sep 09 21:21:59.648282 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:21:59.648060    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:21:59.648282 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:21:59.648282 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:21:59.648282 ip-10-0-140-132 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerName="main" containerID="cri-o://c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e"
Sep 09 21:21:59.648282 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:21:59.648098    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerName="main" containerID="cri-o://c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e" gracePeriod=60
Sep 09 21:21:59.867615 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:21:59.867567    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerName="llm-d-routing-sidecar" containerID="cri-o://22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8" gracePeriod=60
Sep 09 21:21:59.960467 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:21:59.960378    2583 generic.go:356] "Generic (PLEG): container finished" podID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerID="c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e" exitCode=0
Sep 09 21:21:59.960467 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:21:59.960451    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" event={"ID":"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4","Type":"ContainerDied","Data":"c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e"}
Sep 09 21:22:00.130716 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.130691    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:22:00.218037 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.217940    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72t9z\" (UniqueName: \"kubernetes.io/projected/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-kube-api-access-72t9z\") pod \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") "
Sep 09 21:22:00.218037 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.218021    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-model-cache\") pod \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") "
Sep 09 21:22:00.218269 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.218099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tls-certs\") pod \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") "
Sep 09 21:22:00.218269 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.218183    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-dshm\") pod \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") "
Sep 09 21:22:00.218269 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.218213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tmp-dir\") pod \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") "
Sep 09 21:22:00.218269 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.218242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-home\") pod \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\" (UID: \"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4\") "
Sep 09 21:22:00.218433 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.218322    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" (UID: "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:00.218550 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.218522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-home" (OuterVolumeSpecName: "home") pod "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" (UID: "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:00.218678 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.218589    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" (UID: "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:00.218678 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.218612    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.221015 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.220983    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-kube-api-access-72t9z" (OuterVolumeSpecName: "kube-api-access-72t9z") pod "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" (UID: "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4"). InnerVolumeSpecName "kube-api-access-72t9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:00.221015 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.221002    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-dshm" (OuterVolumeSpecName: "dshm") pod "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" (UID: "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:00.221181 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.221036    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" (UID: "3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:00.319706 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.319664    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.319706 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.319697    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.319706 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.319711    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.319939 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.319722    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72t9z\" (UniqueName: \"kubernetes.io/projected/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-kube-api-access-72t9z\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.319939 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.319734    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.967382 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.967346    2583 generic.go:356] "Generic (PLEG): container finished" podID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerID="22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8" exitCode=0
Sep 09 21:22:00.967806 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.967401    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"
Sep 09 21:22:00.967806 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.967440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" event={"ID":"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4","Type":"ContainerDied","Data":"22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8"}
Sep 09 21:22:00.967806 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.967503    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm" event={"ID":"3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4","Type":"ContainerDied","Data":"cd74385f84e29d9bd80c75fd67309a20a89326ebc983751a794c1b03665eb99c"}
Sep 09 21:22:00.967806 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.967537    2583 scope.go:117] "RemoveContainer" containerID="c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e"
Sep 09 21:22:00.977624 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.977603    2583 scope.go:117] "RemoveContainer" containerID="22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8"
Sep 09 21:22:00.985682 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.985662    2583 scope.go:117] "RemoveContainer" containerID="c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e"
Sep 09 21:22:00.986028 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:22:00.986003    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e\": container with ID starting with c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e not found: ID does not exist" containerID="c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e"
Sep 09 21:22:00.986156 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.986033    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e"} err="failed to get container status \"c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e\": rpc error: code = NotFound desc = could not find container \"c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e\": container with ID starting with c5d6d1b075d99f1925a15b15375f152512426f79302a376b0adc1c4740fc381e not found: ID does not exist"
Sep 09 21:22:00.986156 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.986053    2583 scope.go:117] "RemoveContainer" containerID="22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8"
Sep 09 21:22:00.986385 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:22:00.986367    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8\": container with ID starting with 22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8 not found: ID does not exist" containerID="22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8"
Sep 09 21:22:00.986438 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.986391    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8"} err="failed to get container status \"22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8\": rpc error: code = NotFound desc = could not find container \"22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8\": container with ID starting with 22fe0e91ca3700d32f62cb6fd439cb934a02297a5b43c68dc20fdf141e4b94c8 not found: ID does not exist"
Sep 09 21:22:00.991271 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.991240    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"]
Sep 09 21:22:00.997353 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:00.997324    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mv8nm"]
Sep 09 21:22:01.865574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:22:01.865544    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" path="/var/lib/kubelet/pods/3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4/volumes"
Sep 09 21:23:15.061924 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.061886    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"]
Sep 09 21:23:15.062380 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.062291    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerName="llm-d-routing-sidecar"
Sep 09 21:23:15.062380 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.062304    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerName="llm-d-routing-sidecar"
Sep 09 21:23:15.062380 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.062314    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerName="main"
Sep 09 21:23:15.062380 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.062319    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerName="main"
Sep 09 21:23:15.062513 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.062388    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerName="llm-d-routing-sidecar"
Sep 09 21:23:15.062513 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.062400    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b01a2f4-e42f-4cdc-b56c-b0ac27dc8fa4" containerName="main"
Sep 09 21:23:15.065311 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.065296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.070438 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.070406    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 09 21:23:15.070603 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.070456    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 09 21:23:15.070603 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.070409    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-lqht4\""
Sep 09 21:23:15.070603 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.070512    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 09 21:23:15.080021 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.079994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n42d\" (UniqueName: \"kubernetes.io/projected/1fca2fbe-c792-4da7-98b3-4a83cef582f2-kube-api-access-9n42d\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.080210 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.080043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.080210 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.080157    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"]
Sep 09 21:23:15.080210 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.080198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.080365 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.080288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.080365 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.080322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.080365 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.080345    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.181010 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.180964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.181254 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.181024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.181254 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.181048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.181254 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.181064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.181254 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.181087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n42d\" (UniqueName: \"kubernetes.io/projected/1fca2fbe-c792-4da7-98b3-4a83cef582f2-kube-api-access-9n42d\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.181254 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.181164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.181537 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.181513    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.181591 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.181538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.181591 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.181552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.183531 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.183502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.183768 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.183752    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.190369 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.190346    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n42d\" (UniqueName: \"kubernetes.io/projected/1fca2fbe-c792-4da7-98b3-4a83cef582f2-kube-api-access-9n42d\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.376798 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.376701    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:15.653294 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:15.653205    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"]
Sep 09 21:23:15.656895 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:23:15.656862    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fca2fbe_c792_4da7_98b3_4a83cef582f2.slice/crio-fab982b52be60857968cc9dbbae4cf7eeeb5dbd63364264cdb5d0a234615aa2a WatchSource:0}: Error finding container fab982b52be60857968cc9dbbae4cf7eeeb5dbd63364264cdb5d0a234615aa2a: Status 404 returned error can't find the container with id fab982b52be60857968cc9dbbae4cf7eeeb5dbd63364264cdb5d0a234615aa2a
Sep 09 21:23:16.318410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:16.318372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj" event={"ID":"1fca2fbe-c792-4da7-98b3-4a83cef582f2","Type":"ContainerStarted","Data":"aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f"}
Sep 09 21:23:16.318811 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:16.318438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj" event={"ID":"1fca2fbe-c792-4da7-98b3-4a83cef582f2","Type":"ContainerStarted","Data":"fab982b52be60857968cc9dbbae4cf7eeeb5dbd63364264cdb5d0a234615aa2a"}
Sep 09 21:23:16.355359 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:16.355297    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj" podStartSLOduration=1.355278452 podStartE2EDuration="1.355278452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:16.353706675 +0000 UTC m=+1834.978180617" watchObservedRunningTime="2026-09-09 21:23:16.355278452 +0000 UTC m=+1834.979752393"
Sep 09 21:23:16.377816 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:16.377778    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:25.376933 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:25.376900    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:25.389322 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:25.389287    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:23:26.375968 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:23:26.375939    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:24:10.961863 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:10.961786    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"]
Sep 09 21:24:10.971769 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:24:10.971729    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:24:10.971769 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:24:10.971769 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:24:10.971769 ip-10-0-140-132 kubenswrapper[2583]:  > containerID="aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f" cmd=["/bin/sleep","15"]
Sep 09 21:24:10.972031 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:10.971792    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj" message="" err=<
Sep 09 21:24:10.972031 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:24:10.972031 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:24:10.972031 ip-10-0-140-132 kubenswrapper[2583]:  >
Sep 09 21:24:10.972031 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:24:10.971811    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:24:10.972031 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:24:10.972031 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:24:10.972031 ip-10-0-140-132 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj" podUID="1fca2fbe-c792-4da7-98b3-4a83cef582f2" containerName="main" containerID="cri-o://aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f"
Sep 09 21:24:10.972031 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:10.971856    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj" podUID="1fca2fbe-c792-4da7-98b3-4a83cef582f2" containerName="main" containerID="cri-o://aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f" gracePeriod=60
Sep 09 21:24:11.324781 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.324750    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:24:11.388769 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.388735    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tls-certs\") pod \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") "
Sep 09 21:24:11.388769 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.388775    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-home\") pod \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") "
Sep 09 21:24:11.389004 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.388821    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-model-cache\") pod \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") "
Sep 09 21:24:11.389004 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.388990    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n42d\" (UniqueName: \"kubernetes.io/projected/1fca2fbe-c792-4da7-98b3-4a83cef582f2-kube-api-access-9n42d\") pod \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") "
Sep 09 21:24:11.389099 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.389065    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fca2fbe-c792-4da7-98b3-4a83cef582f2" (UID: "1fca2fbe-c792-4da7-98b3-4a83cef582f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:11.389099 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.389072    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-home" (OuterVolumeSpecName: "home") pod "1fca2fbe-c792-4da7-98b3-4a83cef582f2" (UID: "1fca2fbe-c792-4da7-98b3-4a83cef582f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:11.389099 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.389098    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-dshm\") pod \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") "
Sep 09 21:24:11.389230 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.389177    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tmp-dir\") pod \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\" (UID: \"1fca2fbe-c792-4da7-98b3-4a83cef582f2\") "
Sep 09 21:24:11.389488 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.389445    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fca2fbe-c792-4da7-98b3-4a83cef582f2" (UID: "1fca2fbe-c792-4da7-98b3-4a83cef582f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:11.389611 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.389451    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.389611 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.389518    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.391379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.391348    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fca2fbe-c792-4da7-98b3-4a83cef582f2" (UID: "1fca2fbe-c792-4da7-98b3-4a83cef582f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:11.391489 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.391386    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-dshm" (OuterVolumeSpecName: "dshm") pod "1fca2fbe-c792-4da7-98b3-4a83cef582f2" (UID: "1fca2fbe-c792-4da7-98b3-4a83cef582f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:11.391489 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.391413    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fca2fbe-c792-4da7-98b3-4a83cef582f2-kube-api-access-9n42d" (OuterVolumeSpecName: "kube-api-access-9n42d") pod "1fca2fbe-c792-4da7-98b3-4a83cef582f2" (UID: "1fca2fbe-c792-4da7-98b3-4a83cef582f2"). InnerVolumeSpecName "kube-api-access-9n42d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:11.490614 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.490516    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.490614 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.490549    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n42d\" (UniqueName: \"kubernetes.io/projected/1fca2fbe-c792-4da7-98b3-4a83cef582f2-kube-api-access-9n42d\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.490614 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.490563    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.490614 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.490575    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fca2fbe-c792-4da7-98b3-4a83cef582f2-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.570995 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.570959    2583 generic.go:356] "Generic (PLEG): container finished" podID="1fca2fbe-c792-4da7-98b3-4a83cef582f2" containerID="aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f" exitCode=0
Sep 09 21:24:11.571215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.571006    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"
Sep 09 21:24:11.571215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.571043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj" event={"ID":"1fca2fbe-c792-4da7-98b3-4a83cef582f2","Type":"ContainerDied","Data":"aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f"}
Sep 09 21:24:11.571215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.571089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj" event={"ID":"1fca2fbe-c792-4da7-98b3-4a83cef582f2","Type":"ContainerDied","Data":"fab982b52be60857968cc9dbbae4cf7eeeb5dbd63364264cdb5d0a234615aa2a"}
Sep 09 21:24:11.571215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.571141    2583 scope.go:117] "RemoveContainer" containerID="aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f"
Sep 09 21:24:11.583016 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.582955    2583 scope.go:117] "RemoveContainer" containerID="aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f"
Sep 09 21:24:11.583600 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:24:11.583575    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f\": container with ID starting with aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f not found: ID does not exist" containerID="aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f"
Sep 09 21:24:11.583696 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.583610    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f"} err="failed to get container status \"aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f\": rpc error: code = NotFound desc = could not find container \"aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f\": container with ID starting with aa7cc9820b227193e9cf1c17774e110afdaceea66bab0f3d2ed85985eb63450f not found: ID does not exist"
Sep 09 21:24:11.602308 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.602284    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"]
Sep 09 21:24:11.606742 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.606723    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xjvjj"]
Sep 09 21:24:11.868757 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:24:11.868720    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fca2fbe-c792-4da7-98b3-4a83cef582f2" path="/var/lib/kubelet/pods/1fca2fbe-c792-4da7-98b3-4a83cef582f2/volumes"
Sep 09 21:25:26.016042 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.014584    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"]
Sep 09 21:25:26.016042 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.015510    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fca2fbe-c792-4da7-98b3-4a83cef582f2" containerName="main"
Sep 09 21:25:26.016042 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.015533    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fca2fbe-c792-4da7-98b3-4a83cef582f2" containerName="main"
Sep 09 21:25:26.016042 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.015673    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fca2fbe-c792-4da7-98b3-4a83cef582f2" containerName="main"
Sep 09 21:25:26.023120 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.023072    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.028734 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.028667    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 09 21:25:26.029037 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.029020    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:25:26.029349 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.029331    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 09 21:25:26.030890 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.030628    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 09 21:25:26.030965 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.030907    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-dndtl\""
Sep 09 21:25:26.031072 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.031049    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 09 21:25:26.033548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.033522    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"]
Sep 09 21:25:26.169387 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.169351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-dshm\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.169387 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.169387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.169624 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.169416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-home\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.169624 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.169445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.169624 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.169489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9dc\" (UniqueName: \"kubernetes.io/projected/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kube-api-access-mm9dc\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.169624 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.169604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-model-cache\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.169757 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.169632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.169757 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.169652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270251 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-dshm\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270251 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270200    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270251 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-home\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270528 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270528 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9dc\" (UniqueName: \"kubernetes.io/projected/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kube-api-access-mm9dc\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270528 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-model-cache\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270528 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270528 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270800 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270748    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270855 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-home\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270855 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.270962 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.270894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-model-cache\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.271139 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.271071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.272852 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.272828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-dshm\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.272978 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.272954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.279327 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.279304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9dc\" (UniqueName: \"kubernetes.io/projected/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kube-api-access-mm9dc\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-n48zx\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.336071 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.336037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:26.622716 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.622687    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"]
Sep 09 21:25:26.625050 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:25:26.625025    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80f3093_a6b9_4958_b7b4_491fbd3d1be9.slice/crio-ba320b27423c6a1d996ab2086251f576d49b5879c0ed1bc1b890d22264f99943 WatchSource:0}: Error finding container ba320b27423c6a1d996ab2086251f576d49b5879c0ed1bc1b890d22264f99943: Status 404 returned error can't find the container with id ba320b27423c6a1d996ab2086251f576d49b5879c0ed1bc1b890d22264f99943
Sep 09 21:25:26.915201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.915084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" event={"ID":"c80f3093-a6b9-4958-b7b4-491fbd3d1be9","Type":"ContainerStarted","Data":"1cdbeff0d234d7211c01ee09dc5a0b0a089456b657b3980efae98c85ad6a6762"}
Sep 09 21:25:26.915201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:26.915166    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" event={"ID":"c80f3093-a6b9-4958-b7b4-491fbd3d1be9","Type":"ContainerStarted","Data":"ba320b27423c6a1d996ab2086251f576d49b5879c0ed1bc1b890d22264f99943"}
Sep 09 21:25:29.931586 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:29.931551    2583 generic.go:356] "Generic (PLEG): container finished" podID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" containerID="1cdbeff0d234d7211c01ee09dc5a0b0a089456b657b3980efae98c85ad6a6762" exitCode=0
Sep 09 21:25:29.932137 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:29.931627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" event={"ID":"c80f3093-a6b9-4958-b7b4-491fbd3d1be9","Type":"ContainerDied","Data":"1cdbeff0d234d7211c01ee09dc5a0b0a089456b657b3980efae98c85ad6a6762"}
Sep 09 21:25:30.939232 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:30.939196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" event={"ID":"c80f3093-a6b9-4958-b7b4-491fbd3d1be9","Type":"ContainerStarted","Data":"8d90446ccfc9bcad92d89f8f5dde04b6cb0d01e109f71a5ae03b17af9ac67a0d"}
Sep 09 21:25:30.964488 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:30.964435    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" podStartSLOduration=3.735210758 podStartE2EDuration="5.964417374s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.229206616s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:25:30.962811136 +0000 UTC m=+1969.587285076" watchObservedRunningTime="2026-09-09 21:25:30.964417374 +0000 UTC m=+1969.588891314"
Sep 09 21:25:31.336476 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:31.336434    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:36.336330 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:36.336232    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:36.348864 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:36.348841    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:25:36.977135 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:25:36.977083    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:27:00.593454 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.593421    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"]
Sep 09 21:27:00.597136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.597099    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.600607 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.600585    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 09 21:27:00.600738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.600666    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-kwhkw\""
Sep 09 21:27:00.601879 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.601844    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 09 21:27:00.601998 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.601884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 09 21:27:00.613718 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.613697    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"]
Sep 09 21:27:00.627565 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.627527    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-dshm\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.627836 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.627797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-model-cache\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.627958 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.627840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcjfr\" (UniqueName: \"kubernetes.io/projected/7405e754-9506-49fd-bd6b-52bde00cb51c-kube-api-access-vcjfr\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.627958 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.627901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-home\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.628094 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.627972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7405e754-9506-49fd-bd6b-52bde00cb51c-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.628094 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.628018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.728696 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.728661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-dshm\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.728696 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.728695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-model-cache\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.728959 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.728711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcjfr\" (UniqueName: \"kubernetes.io/projected/7405e754-9506-49fd-bd6b-52bde00cb51c-kube-api-access-vcjfr\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.728959 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.728729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-home\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.728959 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.728866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7405e754-9506-49fd-bd6b-52bde00cb51c-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.728959 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.728923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.729230 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.729158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-model-cache\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.729440 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.729409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.729568 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.729556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-home\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.731411 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.731385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-dshm\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.731521 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.731502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7405e754-9506-49fd-bd6b-52bde00cb51c-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.739554 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.739531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcjfr\" (UniqueName: \"kubernetes.io/projected/7405e754-9506-49fd-bd6b-52bde00cb51c-kube-api-access-vcjfr\") pod \"decom-v1-kserve-6cb797fb75-thkfw\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:00.907880 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:00.907777    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:01.193735 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:01.193710    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"]
Sep 09 21:27:01.195916 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:27:01.195875    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7405e754_9506_49fd_bd6b_52bde00cb51c.slice/crio-901d01d147f5264168cbe15457917033d349bea469575ab4eeaee1b7ab1c5d94 WatchSource:0}: Error finding container 901d01d147f5264168cbe15457917033d349bea469575ab4eeaee1b7ab1c5d94: Status 404 returned error can't find the container with id 901d01d147f5264168cbe15457917033d349bea469575ab4eeaee1b7ab1c5d94
Sep 09 21:27:01.197989 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:01.197962    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:27:01.355522 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:01.355491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw" event={"ID":"7405e754-9506-49fd-bd6b-52bde00cb51c","Type":"ContainerStarted","Data":"901d01d147f5264168cbe15457917033d349bea469575ab4eeaee1b7ab1c5d94"}
Sep 09 21:27:02.363005 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:02.362963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw" event={"ID":"7405e754-9506-49fd-bd6b-52bde00cb51c","Type":"ContainerStarted","Data":"1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215"}
Sep 09 21:27:02.385337 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:02.385280    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw" podStartSLOduration=1.382370807 podStartE2EDuration="2.385260097s" totalImagesPullingTime="1.00288929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:27:02.383992204 +0000 UTC m=+2061.008466144" watchObservedRunningTime="2026-09-09 21:27:02.385260097 +0000 UTC m=+2061.009734051"
Sep 09 21:27:02.908320 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:02.908226    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:10.908924 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:10.908885    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:10.921904 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:10.921876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:11.419679 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:11.419648    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:27:40.276533 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.276479    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"]
Sep 09 21:27:40.286782 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:27:40.286735    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:27:40.286782 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:27:40.286782 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:27:40.286782 ip-10-0-140-132 kubenswrapper[2583]:  > containerID="8d90446ccfc9bcad92d89f8f5dde04b6cb0d01e109f71a5ae03b17af9ac67a0d" cmd=["/bin/sleep","15"]
Sep 09 21:27:40.287338 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.286803    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" message="" err=<
Sep 09 21:27:40.287338 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:27:40.287338 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:27:40.287338 ip-10-0-140-132 kubenswrapper[2583]:  >
Sep 09 21:27:40.287338 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:27:40.286820    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:27:40.287338 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:27:40.287338 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:27:40.287338 ip-10-0-140-132 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" podUID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" containerName="main" containerID="cri-o://8d90446ccfc9bcad92d89f8f5dde04b6cb0d01e109f71a5ae03b17af9ac67a0d"
Sep 09 21:27:40.287338 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.286877    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" podUID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" containerName="main" containerID="cri-o://8d90446ccfc9bcad92d89f8f5dde04b6cb0d01e109f71a5ae03b17af9ac67a0d" gracePeriod=60
Sep 09 21:27:40.541222 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.541133    2583 generic.go:356] "Generic (PLEG): container finished" podID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" containerID="8d90446ccfc9bcad92d89f8f5dde04b6cb0d01e109f71a5ae03b17af9ac67a0d" exitCode=0
Sep 09 21:27:40.541222 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.541184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" event={"ID":"c80f3093-a6b9-4958-b7b4-491fbd3d1be9","Type":"ContainerDied","Data":"8d90446ccfc9bcad92d89f8f5dde04b6cb0d01e109f71a5ae03b17af9ac67a0d"}
Sep 09 21:27:40.644005 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.643972    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:27:40.799439 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799412    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kserve-provision-location\") pod \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") "
Sep 09 21:27:40.799638 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799452    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-home\") pod \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") "
Sep 09 21:27:40.799638 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799476    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm9dc\" (UniqueName: \"kubernetes.io/projected/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kube-api-access-mm9dc\") pod \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") "
Sep 09 21:27:40.799638 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tmp-dir\") pod \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") "
Sep 09 21:27:40.799638 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799574    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-cabundle-cert\") pod \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") "
Sep 09 21:27:40.799638 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799599    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-model-cache\") pod \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") "
Sep 09 21:27:40.799638 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799628    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tls-certs\") pod \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") "
Sep 09 21:27:40.799949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799694    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-dshm\") pod \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\" (UID: \"c80f3093-a6b9-4958-b7b4-491fbd3d1be9\") "
Sep 09 21:27:40.799949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799783    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-home" (OuterVolumeSpecName: "home") pod "c80f3093-a6b9-4958-b7b4-491fbd3d1be9" (UID: "c80f3093-a6b9-4958-b7b4-491fbd3d1be9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.799949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799916    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c80f3093-a6b9-4958-b7b4-491fbd3d1be9" (UID: "c80f3093-a6b9-4958-b7b4-491fbd3d1be9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.799949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.799933    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c80f3093-a6b9-4958-b7b4-491fbd3d1be9" (UID: "c80f3093-a6b9-4958-b7b4-491fbd3d1be9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.800460 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.800024    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c80f3093-a6b9-4958-b7b4-491fbd3d1be9" (UID: "c80f3093-a6b9-4958-b7b4-491fbd3d1be9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:27:40.800460 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.800233    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.800460 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.800250    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.800460 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.800260    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.800460 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.800268    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.802120 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.802083    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-dshm" (OuterVolumeSpecName: "dshm") pod "c80f3093-a6b9-4958-b7b4-491fbd3d1be9" (UID: "c80f3093-a6b9-4958-b7b4-491fbd3d1be9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.802590 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.802572    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c80f3093-a6b9-4958-b7b4-491fbd3d1be9" (UID: "c80f3093-a6b9-4958-b7b4-491fbd3d1be9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:27:40.802590 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.802581    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kube-api-access-mm9dc" (OuterVolumeSpecName: "kube-api-access-mm9dc") pod "c80f3093-a6b9-4958-b7b4-491fbd3d1be9" (UID: "c80f3093-a6b9-4958-b7b4-491fbd3d1be9"). InnerVolumeSpecName "kube-api-access-mm9dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:27:40.828586 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.828541    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c80f3093-a6b9-4958-b7b4-491fbd3d1be9" (UID: "c80f3093-a6b9-4958-b7b4-491fbd3d1be9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.901465 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.901419    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.901465 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.901456    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9dc\" (UniqueName: \"kubernetes.io/projected/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-kube-api-access-mm9dc\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.901465 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.901466    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.901465 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:40.901476    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80f3093-a6b9-4958-b7b4-491fbd3d1be9-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:27:41.548448 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:41.548419    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"
Sep 09 21:27:41.548958 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:41.548416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx" event={"ID":"c80f3093-a6b9-4958-b7b4-491fbd3d1be9","Type":"ContainerDied","Data":"ba320b27423c6a1d996ab2086251f576d49b5879c0ed1bc1b890d22264f99943"}
Sep 09 21:27:41.548958 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:41.548551    2583 scope.go:117] "RemoveContainer" containerID="8d90446ccfc9bcad92d89f8f5dde04b6cb0d01e109f71a5ae03b17af9ac67a0d"
Sep 09 21:27:41.558995 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:41.558973    2583 scope.go:117] "RemoveContainer" containerID="1cdbeff0d234d7211c01ee09dc5a0b0a089456b657b3980efae98c85ad6a6762"
Sep 09 21:27:41.575857 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:41.575827    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"]
Sep 09 21:27:41.581636 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:41.581612    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-n48zx"]
Sep 09 21:27:41.867051 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:27:41.866967    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" path="/var/lib/kubelet/pods/c80f3093-a6b9-4958-b7b4-491fbd3d1be9/volumes"
Sep 09 21:28:11.342379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:11.342342    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"]
Sep 09 21:28:26.363125 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.363062    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw" podUID="7405e754-9506-49fd-bd6b-52bde00cb51c" containerName="main" containerID="cri-o://1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215" gracePeriod=45
Sep 09 21:28:26.407920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.407883    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw" podUID="7405e754-9506-49fd-bd6b-52bde00cb51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 09 21:28:26.616895 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.616818    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:28:26.751605 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.751570    2583 generic.go:356] "Generic (PLEG): container finished" podID="7405e754-9506-49fd-bd6b-52bde00cb51c" containerID="1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215" exitCode=0
Sep 09 21:28:26.751772 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.751623    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"
Sep 09 21:28:26.751772 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.751654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw" event={"ID":"7405e754-9506-49fd-bd6b-52bde00cb51c","Type":"ContainerDied","Data":"1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215"}
Sep 09 21:28:26.751772 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.751712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw" event={"ID":"7405e754-9506-49fd-bd6b-52bde00cb51c","Type":"ContainerDied","Data":"901d01d147f5264168cbe15457917033d349bea469575ab4eeaee1b7ab1c5d94"}
Sep 09 21:28:26.751772 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.751744    2583 scope.go:117] "RemoveContainer" containerID="1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215"
Sep 09 21:28:26.761027 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.761003    2583 scope.go:117] "RemoveContainer" containerID="1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215"
Sep 09 21:28:26.761334 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:28:26.761313    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215\": container with ID starting with 1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215 not found: ID does not exist" containerID="1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215"
Sep 09 21:28:26.761417 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.761346    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215"} err="failed to get container status \"1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215\": rpc error: code = NotFound desc = could not find container \"1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215\": container with ID starting with 1162e3d7171a3fe6d7d0f2421694d1671cd7c6740abc2dd6f07a718534aa1215 not found: ID does not exist"
Sep 09 21:28:26.809279 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcjfr\" (UniqueName: \"kubernetes.io/projected/7405e754-9506-49fd-bd6b-52bde00cb51c-kube-api-access-vcjfr\") pod \"7405e754-9506-49fd-bd6b-52bde00cb51c\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") "
Sep 09 21:28:26.809279 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7405e754-9506-49fd-bd6b-52bde00cb51c-tls-certs\") pod \"7405e754-9506-49fd-bd6b-52bde00cb51c\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") "
Sep 09 21:28:26.809552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809412    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-home\") pod \"7405e754-9506-49fd-bd6b-52bde00cb51c\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") "
Sep 09 21:28:26.809552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809443    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-tmp-dir\") pod \"7405e754-9506-49fd-bd6b-52bde00cb51c\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") "
Sep 09 21:28:26.809552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809467    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-dshm\") pod \"7405e754-9506-49fd-bd6b-52bde00cb51c\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") "
Sep 09 21:28:26.809552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809496    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-model-cache\") pod \"7405e754-9506-49fd-bd6b-52bde00cb51c\" (UID: \"7405e754-9506-49fd-bd6b-52bde00cb51c\") "
Sep 09 21:28:26.809784 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809684    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-home" (OuterVolumeSpecName: "home") pod "7405e754-9506-49fd-bd6b-52bde00cb51c" (UID: "7405e754-9506-49fd-bd6b-52bde00cb51c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:28:26.809784 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7405e754-9506-49fd-bd6b-52bde00cb51c" (UID: "7405e754-9506-49fd-bd6b-52bde00cb51c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:28:26.809908 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809821    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:28:26.809908 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.809832    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:28:26.810329 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.810298    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7405e754-9506-49fd-bd6b-52bde00cb51c" (UID: "7405e754-9506-49fd-bd6b-52bde00cb51c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:28:26.811806 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.811766    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-dshm" (OuterVolumeSpecName: "dshm") pod "7405e754-9506-49fd-bd6b-52bde00cb51c" (UID: "7405e754-9506-49fd-bd6b-52bde00cb51c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:28:26.811894 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.811827    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7405e754-9506-49fd-bd6b-52bde00cb51c-kube-api-access-vcjfr" (OuterVolumeSpecName: "kube-api-access-vcjfr") pod "7405e754-9506-49fd-bd6b-52bde00cb51c" (UID: "7405e754-9506-49fd-bd6b-52bde00cb51c"). InnerVolumeSpecName "kube-api-access-vcjfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:28:26.811949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.811888    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7405e754-9506-49fd-bd6b-52bde00cb51c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7405e754-9506-49fd-bd6b-52bde00cb51c" (UID: "7405e754-9506-49fd-bd6b-52bde00cb51c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:28:26.910381 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.910294    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:28:26.910381 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.910323    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcjfr\" (UniqueName: \"kubernetes.io/projected/7405e754-9506-49fd-bd6b-52bde00cb51c-kube-api-access-vcjfr\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:28:26.910381 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.910333    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7405e754-9506-49fd-bd6b-52bde00cb51c-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:28:26.910381 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:26.910342    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7405e754-9506-49fd-bd6b-52bde00cb51c-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:28:27.078026 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:27.077992    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"]
Sep 09 21:28:27.087415 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:27.087383    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-thkfw"]
Sep 09 21:28:27.864925 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:27.864878    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7405e754-9506-49fd-bd6b-52bde00cb51c" path="/var/lib/kubelet/pods/7405e754-9506-49fd-bd6b-52bde00cb51c/volumes"
Sep 09 21:28:53.524600 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.524554    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"]
Sep 09 21:28:53.524962 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.524939    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" containerName="storage-initializer"
Sep 09 21:28:53.524962 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.524952    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" containerName="storage-initializer"
Sep 09 21:28:53.524962 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.524960    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" containerName="main"
Sep 09 21:28:53.525072 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.524966    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" containerName="main"
Sep 09 21:28:53.525072 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.524976    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7405e754-9506-49fd-bd6b-52bde00cb51c" containerName="main"
Sep 09 21:28:53.525072 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.524984    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7405e754-9506-49fd-bd6b-52bde00cb51c" containerName="main"
Sep 09 21:28:53.525072 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.525040    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80f3093-a6b9-4958-b7b4-491fbd3d1be9" containerName="main"
Sep 09 21:28:53.525072 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.525048    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7405e754-9506-49fd-bd6b-52bde00cb51c" containerName="main"
Sep 09 21:28:53.528242 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.528222    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.531822 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.531802    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 09 21:28:53.531940 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.531870    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 09 21:28:53.533066 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.533041    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:28:53.533201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.533046    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 09 21:28:53.533201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.533134    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-zkt28\""
Sep 09 21:28:53.533201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.533187    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 09 21:28:53.543096 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.543068    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"]
Sep 09 21:28:53.654490 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.654457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.654673 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.654497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.654673 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.654563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.654673 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.654624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzn6\" (UniqueName: \"kubernetes.io/projected/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kube-api-access-nzzn6\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.654673 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.654668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-home\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.654844 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.654698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.654844 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.654714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.654844 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.654754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-dshm\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.755762 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.755720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.755762 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.755766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzn6\" (UniqueName: \"kubernetes.io/projected/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kube-api-access-nzzn6\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756005 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.755794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-home\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756005 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.755812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756005 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.755827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756005 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.755847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-dshm\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756005 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.755898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756005 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.755929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756369 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.756336    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756514 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.756482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-home\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756514 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.756500    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756623 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.756560    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.756988 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.756966    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.758349 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.758326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-dshm\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.758658 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.758640    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.766780 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.766755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzn6\" (UniqueName: \"kubernetes.io/projected/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kube-api-access-nzzn6\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:53.839192 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:53.839160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:28:54.138213 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:54.138186    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"]
Sep 09 21:28:54.141075 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:28:54.141050    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcc1c1c_54e1_42af_a483_a31e3d6c6f2d.slice/crio-46ff42a861082ac94225966eb558e8ca5133a4ad242a918e4c892892412052ff WatchSource:0}: Error finding container 46ff42a861082ac94225966eb558e8ca5133a4ad242a918e4c892892412052ff: Status 404 returned error can't find the container with id 46ff42a861082ac94225966eb558e8ca5133a4ad242a918e4c892892412052ff
Sep 09 21:28:54.885285 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:54.885245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" event={"ID":"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d","Type":"ContainerStarted","Data":"accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8"}
Sep 09 21:28:54.885666 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:54.885301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" event={"ID":"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d","Type":"ContainerStarted","Data":"46ff42a861082ac94225966eb558e8ca5133a4ad242a918e4c892892412052ff"}
Sep 09 21:28:56.895354 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:56.895312    2583 generic.go:356] "Generic (PLEG): container finished" podID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerID="accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8" exitCode=0
Sep 09 21:28:56.895760 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:56.895390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" event={"ID":"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d","Type":"ContainerDied","Data":"accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8"}
Sep 09 21:28:57.901873 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:57.901838    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" event={"ID":"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d","Type":"ContainerStarted","Data":"9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262"}
Sep 09 21:28:57.931167 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:57.931079    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" podStartSLOduration=2.628781337 podStartE2EDuration="4.93105955s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.302278213s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:28:57.928735001 +0000 UTC m=+2176.553208942" watchObservedRunningTime="2026-09-09 21:28:57.93105955 +0000 UTC m=+2176.555533491"
Sep 09 21:28:58.840201 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:28:58.840170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:29:03.839679 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:03.839632    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:29:03.852462 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:03.852428    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:29:03.949797 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:03.949768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:29:22.897478 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.897446    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"]
Sep 09 21:29:22.901018 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.900992    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:22.906236 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.906216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-gnlgb\""
Sep 09 21:29:22.906386 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.906216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 09 21:29:22.906454 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.906225    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 09 21:29:22.906969 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.906946    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 09 21:29:22.914698 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.914673    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"]
Sep 09 21:29:22.923638 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.923608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fe5d56-9133-4322-9896-60e3461216dc-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:22.923782 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.923655    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-home\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:22.923782 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.923673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:22.923782 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.923690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29b9f\" (UniqueName: \"kubernetes.io/projected/c7fe5d56-9133-4322-9896-60e3461216dc-kube-api-access-29b9f\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:22.923949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.923786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-dshm\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:22.923949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:22.923855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.024369 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.024337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-dshm\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.024584 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.024392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.024584 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.024453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fe5d56-9133-4322-9896-60e3461216dc-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.024584 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.024475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-home\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.024584 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.024497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.024584 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.024517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29b9f\" (UniqueName: \"kubernetes.io/projected/c7fe5d56-9133-4322-9896-60e3461216dc-kube-api-access-29b9f\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.024896 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.024865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.024972 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.024878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-home\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.024972 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.024947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.027071 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.027041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-dshm\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.027335 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.027318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fe5d56-9133-4322-9896-60e3461216dc-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.057370 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.057345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29b9f\" (UniqueName: \"kubernetes.io/projected/c7fe5d56-9133-4322-9896-60e3461216dc-kube-api-access-29b9f\") pod \"tri-v1-kserve-79f8f98bc5-zgrvm\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.211485 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.211397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:23.492382 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:23.492349    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"]
Sep 09 21:29:23.494207 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:29:23.494181    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fe5d56_9133_4322_9896_60e3461216dc.slice/crio-6e152faecc706aca934ebd2c8744cc739141f875ef2d1518c8f36a5e7058267a WatchSource:0}: Error finding container 6e152faecc706aca934ebd2c8744cc739141f875ef2d1518c8f36a5e7058267a: Status 404 returned error can't find the container with id 6e152faecc706aca934ebd2c8744cc739141f875ef2d1518c8f36a5e7058267a
Sep 09 21:29:24.035245 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:24.035201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm" event={"ID":"c7fe5d56-9133-4322-9896-60e3461216dc","Type":"ContainerStarted","Data":"fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce"}
Sep 09 21:29:24.035703 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:24.035262    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm" event={"ID":"c7fe5d56-9133-4322-9896-60e3461216dc","Type":"ContainerStarted","Data":"6e152faecc706aca934ebd2c8744cc739141f875ef2d1518c8f36a5e7058267a"}
Sep 09 21:29:24.059964 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:24.059908    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm" podStartSLOduration=2.059890897 podStartE2EDuration="2.059890897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:29:24.057618725 +0000 UTC m=+2202.682092669" watchObservedRunningTime="2026-09-09 21:29:24.059890897 +0000 UTC m=+2202.684364864"
Sep 09 21:29:24.212239 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:24.212200    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:33.211638 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:33.211603    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:33.224636 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:33.224610    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:29:34.092714 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:29:34.092682    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:30:22.820117 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:22.820006    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"]
Sep 09 21:30:22.831577 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:30:22.831532    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:30:22.831577 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:30:22.831577 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:30:22.831577 ip-10-0-140-132 kubenswrapper[2583]:  > containerID="9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262" cmd=["/bin/sleep","15"]
Sep 09 21:30:22.831892 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:22.831603    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" message="" err=<
Sep 09 21:30:22.831892 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:30:22.831892 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:30:22.831892 ip-10-0-140-132 kubenswrapper[2583]:  >
Sep 09 21:30:22.831892 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:30:22.831621    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:30:22.831892 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:30:22.831892 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:30:22.831892 ip-10-0-140-132 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" podUID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerName="main" containerID="cri-o://9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262"
Sep 09 21:30:22.831892 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:22.831680    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" podUID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerName="main" containerID="cri-o://9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262" gracePeriod=60
Sep 09 21:30:22.939260 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:22.939229    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" podUID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 09 21:30:23.183569 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.183542    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:30:23.276864 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.276827    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kserve-provision-location\") pod \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") "
Sep 09 21:30:23.277053 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.276884    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tmp-dir\") pod \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") "
Sep 09 21:30:23.277053 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.276905    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-dshm\") pod \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") "
Sep 09 21:30:23.277053 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.276931    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzzn6\" (UniqueName: \"kubernetes.io/projected/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kube-api-access-nzzn6\") pod \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") "
Sep 09 21:30:23.277053 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.276947    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-home\") pod \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") "
Sep 09 21:30:23.277053 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.276965    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-model-cache\") pod \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") "
Sep 09 21:30:23.277053 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.276994    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tls-certs\") pod \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") "
Sep 09 21:30:23.277053 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.277008    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-cabundle-cert\") pod \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\" (UID: \"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d\") "
Sep 09 21:30:23.277452 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.277263    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" (UID: "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.277452 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.277334    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-home" (OuterVolumeSpecName: "home") pod "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" (UID: "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.277548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.277510    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" (UID: "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.277613 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.277558    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" (UID: "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:30:23.279574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.279546    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" (UID: "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:30:23.279789 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.279761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kube-api-access-nzzn6" (OuterVolumeSpecName: "kube-api-access-nzzn6") pod "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" (UID: "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d"). InnerVolumeSpecName "kube-api-access-nzzn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:30:23.279789 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.279779    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-dshm" (OuterVolumeSpecName: "dshm") pod "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" (UID: "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.297787 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.297761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" (UID: "dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.316925 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.316890    2583 generic.go:356] "Generic (PLEG): container finished" podID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerID="9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262" exitCode=0
Sep 09 21:30:23.317073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.316955    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"
Sep 09 21:30:23.317073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.316965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" event={"ID":"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d","Type":"ContainerDied","Data":"9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262"}
Sep 09 21:30:23.317073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.317025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv" event={"ID":"dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d","Type":"ContainerDied","Data":"46ff42a861082ac94225966eb558e8ca5133a4ad242a918e4c892892412052ff"}
Sep 09 21:30:23.317073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.317050    2583 scope.go:117] "RemoveContainer" containerID="9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262"
Sep 09 21:30:23.330422 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.330399    2583 scope.go:117] "RemoveContainer" containerID="accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8"
Sep 09 21:30:23.338958 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.338938    2583 scope.go:117] "RemoveContainer" containerID="9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262"
Sep 09 21:30:23.339345 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:30:23.339322    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262\": container with ID starting with 9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262 not found: ID does not exist" containerID="9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262"
Sep 09 21:30:23.339439 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.339357    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262"} err="failed to get container status \"9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262\": rpc error: code = NotFound desc = could not find container \"9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262\": container with ID starting with 9974e25066eb5fe47641cfe4baa15e4674eae03f27a9a4c4245b2c16d6d2a262 not found: ID does not exist"
Sep 09 21:30:23.339439 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.339381    2583 scope.go:117] "RemoveContainer" containerID="accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8"
Sep 09 21:30:23.339698 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:30:23.339681    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8\": container with ID starting with accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8 not found: ID does not exist" containerID="accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8"
Sep 09 21:30:23.339743 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.339704    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8"} err="failed to get container status \"accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8\": rpc error: code = NotFound desc = could not find container \"accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8\": container with ID starting with accea289cd955553b6079ea2b243676fd19c11f25f854811d904904675e3e1e8 not found: ID does not exist"
Sep 09 21:30:23.348032 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.348006    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"]
Sep 09 21:30:23.353796 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.353730    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-b97zv"]
Sep 09 21:30:23.377961 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.377931    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.378088 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.377975    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.378088 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.377984    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.378088 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.378002    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.378088 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.378013    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.378088 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.378024    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.378088 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.378032    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzzn6\" (UniqueName: \"kubernetes.io/projected/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-kube-api-access-nzzn6\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.378088 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.378040    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.865945 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:30:23.865914    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" path="/var/lib/kubelet/pods/dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d/volumes"
Sep 09 21:31:03.633376 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:03.633340    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"]
Sep 09 21:31:18.653559 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.653514    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm" podUID="c7fe5d56-9133-4322-9896-60e3461216dc" containerName="main" containerID="cri-o://fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce" gracePeriod=45
Sep 09 21:31:18.907305 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.907237    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:31:18.962583 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.962549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fe5d56-9133-4322-9896-60e3461216dc-tls-certs\") pod \"c7fe5d56-9133-4322-9896-60e3461216dc\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") "
Sep 09 21:31:18.962761 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.962597    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29b9f\" (UniqueName: \"kubernetes.io/projected/c7fe5d56-9133-4322-9896-60e3461216dc-kube-api-access-29b9f\") pod \"c7fe5d56-9133-4322-9896-60e3461216dc\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") "
Sep 09 21:31:18.962761 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.962651    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-model-cache\") pod \"c7fe5d56-9133-4322-9896-60e3461216dc\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") "
Sep 09 21:31:18.962761 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.962691    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-tmp-dir\") pod \"c7fe5d56-9133-4322-9896-60e3461216dc\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") "
Sep 09 21:31:18.962929 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.962831    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-home\") pod \"c7fe5d56-9133-4322-9896-60e3461216dc\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") "
Sep 09 21:31:18.962929 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.962876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-dshm\") pod \"c7fe5d56-9133-4322-9896-60e3461216dc\" (UID: \"c7fe5d56-9133-4322-9896-60e3461216dc\") "
Sep 09 21:31:18.963027 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.962929    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7fe5d56-9133-4322-9896-60e3461216dc" (UID: "c7fe5d56-9133-4322-9896-60e3461216dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:18.963027 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.962945    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7fe5d56-9133-4322-9896-60e3461216dc" (UID: "c7fe5d56-9133-4322-9896-60e3461216dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:18.963171 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.963150    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-home" (OuterVolumeSpecName: "home") pod "c7fe5d56-9133-4322-9896-60e3461216dc" (UID: "c7fe5d56-9133-4322-9896-60e3461216dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:18.963257 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.963241    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:31:18.963325 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.963264    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:31:18.963325 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.963276    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:31:18.965029 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.965006    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-dshm" (OuterVolumeSpecName: "dshm") pod "c7fe5d56-9133-4322-9896-60e3461216dc" (UID: "c7fe5d56-9133-4322-9896-60e3461216dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:18.965159 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.965060    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fe5d56-9133-4322-9896-60e3461216dc-kube-api-access-29b9f" (OuterVolumeSpecName: "kube-api-access-29b9f") pod "c7fe5d56-9133-4322-9896-60e3461216dc" (UID: "c7fe5d56-9133-4322-9896-60e3461216dc"). InnerVolumeSpecName "kube-api-access-29b9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:31:18.965159 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:18.965070    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fe5d56-9133-4322-9896-60e3461216dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7fe5d56-9133-4322-9896-60e3461216dc" (UID: "c7fe5d56-9133-4322-9896-60e3461216dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:31:19.064587 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.064552    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fe5d56-9133-4322-9896-60e3461216dc-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:31:19.064587 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.064592    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29b9f\" (UniqueName: \"kubernetes.io/projected/c7fe5d56-9133-4322-9896-60e3461216dc-kube-api-access-29b9f\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:31:19.064774 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.064607    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fe5d56-9133-4322-9896-60e3461216dc-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:31:19.591298 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.591261    2583 generic.go:356] "Generic (PLEG): container finished" podID="c7fe5d56-9133-4322-9896-60e3461216dc" containerID="fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce" exitCode=0
Sep 09 21:31:19.591507 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.591304    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"
Sep 09 21:31:19.591507 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.591329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm" event={"ID":"c7fe5d56-9133-4322-9896-60e3461216dc","Type":"ContainerDied","Data":"fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce"}
Sep 09 21:31:19.591507 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.591369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm" event={"ID":"c7fe5d56-9133-4322-9896-60e3461216dc","Type":"ContainerDied","Data":"6e152faecc706aca934ebd2c8744cc739141f875ef2d1518c8f36a5e7058267a"}
Sep 09 21:31:19.591507 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.591390    2583 scope.go:117] "RemoveContainer" containerID="fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce"
Sep 09 21:31:19.604013 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.603987    2583 scope.go:117] "RemoveContainer" containerID="fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce"
Sep 09 21:31:19.604450 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:31:19.604424    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce\": container with ID starting with fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce not found: ID does not exist" containerID="fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce"
Sep 09 21:31:19.604535 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.604460    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce"} err="failed to get container status \"fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce\": rpc error: code = NotFound desc = could not find container \"fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce\": container with ID starting with fb3916c81c96878c6c97280e5c026352ea2c47da3c76b9127e2999e43e67fbce not found: ID does not exist"
Sep 09 21:31:19.631560 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.631521    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"]
Sep 09 21:31:19.650485 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.650454    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgrvm"]
Sep 09 21:31:19.865375 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:19.865295    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7fe5d56-9133-4322-9896-60e3461216dc" path="/var/lib/kubelet/pods/c7fe5d56-9133-4322-9896-60e3461216dc/volumes"
Sep 09 21:31:21.624730 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.624694    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"]
Sep 09 21:31:21.625239 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.625218    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerName="main"
Sep 09 21:31:21.625323 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.625245    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerName="main"
Sep 09 21:31:21.625323 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.625276    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerName="storage-initializer"
Sep 09 21:31:21.625323 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.625286    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerName="storage-initializer"
Sep 09 21:31:21.625323 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.625303    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fe5d56-9133-4322-9896-60e3461216dc" containerName="main"
Sep 09 21:31:21.625323 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.625311    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fe5d56-9133-4322-9896-60e3461216dc" containerName="main"
Sep 09 21:31:21.625584 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.625396    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7fe5d56-9133-4322-9896-60e3461216dc" containerName="main"
Sep 09 21:31:21.625584 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.625413    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbcc1c1c-54e1-42af-a483-a31e3d6c6f2d" containerName="main"
Sep 09 21:31:21.630093 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.630069    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.633997 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.633974    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 09 21:31:21.635025 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.635000    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 09 21:31:21.635025 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.635011    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 09 21:31:21.635212 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.635003    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-v9wtf\""
Sep 09 21:31:21.635212 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.635003    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 09 21:31:21.635212 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.635000    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:31:21.642359 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.642339    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"]
Sep 09 21:31:21.689827 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.689790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-home\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.689827 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.689831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2734d600-0f4b-439d-9a61-18a3b91a14a1-tls-certs\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.690063 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.689851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbmk5\" (UniqueName: \"kubernetes.io/projected/2734d600-0f4b-439d-9a61-18a3b91a14a1-kube-api-access-wbmk5\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.690063 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.689922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.690063 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.689950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-model-cache\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.690228 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.690060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.690228 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.690126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-dshm\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.690228 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.690146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2734d600-0f4b-439d-9a61-18a3b91a14a1-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.791309 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.791309 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-dshm\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.791548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2734d600-0f4b-439d-9a61-18a3b91a14a1-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.791548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-home\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.791548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2734d600-0f4b-439d-9a61-18a3b91a14a1-tls-certs\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.791548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbmk5\" (UniqueName: \"kubernetes.io/projected/2734d600-0f4b-439d-9a61-18a3b91a14a1-kube-api-access-wbmk5\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.791548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791455    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.791548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-model-cache\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.792027 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.792027 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-home\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.792027 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.791917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.792183 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.792077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2734d600-0f4b-439d-9a61-18a3b91a14a1-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.792183 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.792124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-model-cache\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.793872 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.793852    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-dshm\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.794153 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.794136    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2734d600-0f4b-439d-9a61-18a3b91a14a1-tls-certs\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.801136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.801073    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbmk5\" (UniqueName: \"kubernetes.io/projected/2734d600-0f4b-439d-9a61-18a3b91a14a1-kube-api-access-wbmk5\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-lhbnr\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:21.941864 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:21.941771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:22.226429 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:22.226395    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"]
Sep 09 21:31:22.228509 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:31:22.228480    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2734d600_0f4b_439d_9a61_18a3b91a14a1.slice/crio-d9ceed9e6c1b03baa8bc2ee1e0cde170f4c2b91938e68f74553bd7d6f7d02f63 WatchSource:0}: Error finding container d9ceed9e6c1b03baa8bc2ee1e0cde170f4c2b91938e68f74553bd7d6f7d02f63: Status 404 returned error can't find the container with id d9ceed9e6c1b03baa8bc2ee1e0cde170f4c2b91938e68f74553bd7d6f7d02f63
Sep 09 21:31:22.608571 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:22.608532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" event={"ID":"2734d600-0f4b-439d-9a61-18a3b91a14a1","Type":"ContainerStarted","Data":"267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113"}
Sep 09 21:31:22.608740 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:22.608600    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" event={"ID":"2734d600-0f4b-439d-9a61-18a3b91a14a1","Type":"ContainerStarted","Data":"d9ceed9e6c1b03baa8bc2ee1e0cde170f4c2b91938e68f74553bd7d6f7d02f63"}
Sep 09 21:31:24.620847 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:24.620805    2583 generic.go:356] "Generic (PLEG): container finished" podID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerID="267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113" exitCode=0
Sep 09 21:31:24.621315 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:24.620882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" event={"ID":"2734d600-0f4b-439d-9a61-18a3b91a14a1","Type":"ContainerDied","Data":"267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113"}
Sep 09 21:31:25.627373 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:25.627335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" event={"ID":"2734d600-0f4b-439d-9a61-18a3b91a14a1","Type":"ContainerStarted","Data":"f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456"}
Sep 09 21:31:25.653748 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:25.653698    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" podStartSLOduration=2.350505593 podStartE2EDuration="4.65368306s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.303177467s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:31:25.652384188 +0000 UTC m=+2324.276858129" watchObservedRunningTime="2026-09-09 21:31:25.65368306 +0000 UTC m=+2324.278157002"
Sep 09 21:31:25.942348 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:25.942250    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:31.942550 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:31.942503    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:31.955455 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:31.955427    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:32.672073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:32.671990    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:31:41.689301 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.689266    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"]
Sep 09 21:31:41.692534 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.692516    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.697394 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.697367    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 09 21:31:41.697548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.697372    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 09 21:31:41.697548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.697407    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 09 21:31:41.697548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.697372    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-rs9sr\""
Sep 09 21:31:41.705741 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.705714    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"]
Sep 09 21:31:41.875962 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.875926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tmp-dir\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.876188 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.875983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bwrs\" (UniqueName: \"kubernetes.io/projected/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-kube-api-access-9bwrs\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.876188 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.876157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-dshm\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.876312 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.876196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-home\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.876312 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.876222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-model-cache\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.876312 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.876266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tls-certs\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.977224 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.977130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-dshm\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.977224 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.977172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-home\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.977224 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.977196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-model-cache\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.977525 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.977242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tls-certs\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.977525 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.977269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tmp-dir\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.977525 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.977326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bwrs\" (UniqueName: \"kubernetes.io/projected/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-kube-api-access-9bwrs\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.977697 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.977604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-model-cache\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.977697 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.977654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tmp-dir\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.978254 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.978229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-home\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.979727 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.979710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-dshm\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.980930 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.980914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 09 21:31:41.989304 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.989285    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 09 21:31:41.990436 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.990408    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tls-certs\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:41.998475 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:41.998458    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 09 21:31:42.008532 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:42.008507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bwrs\" (UniqueName: \"kubernetes.io/projected/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-kube-api-access-9bwrs\") pod \"late-v2-kserve-86766546f-448l6\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:42.305656 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:42.305626    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-rs9sr\""
Sep 09 21:31:42.313820 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:42.313790    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:42.601916 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:42.601834    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"]
Sep 09 21:31:42.605674 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:31:42.605642    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43d0ebc_4ae0_4c20_b8fa_9a5803f8737f.slice/crio-358aa76f590d78e99c8a0d051b7c0b2e12f25bec72e172471fa5da51b2ab9959 WatchSource:0}: Error finding container 358aa76f590d78e99c8a0d051b7c0b2e12f25bec72e172471fa5da51b2ab9959: Status 404 returned error can't find the container with id 358aa76f590d78e99c8a0d051b7c0b2e12f25bec72e172471fa5da51b2ab9959
Sep 09 21:31:42.711574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:42.711544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6" event={"ID":"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f","Type":"ContainerStarted","Data":"bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428"}
Sep 09 21:31:42.712057 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:42.711606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6" event={"ID":"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f","Type":"ContainerStarted","Data":"358aa76f590d78e99c8a0d051b7c0b2e12f25bec72e172471fa5da51b2ab9959"}
Sep 09 21:31:42.750763 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:42.750705    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6" podStartSLOduration=1.750686735 podStartE2EDuration="1.750686735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:31:42.748528963 +0000 UTC m=+2341.373002904" watchObservedRunningTime="2026-09-09 21:31:42.750686735 +0000 UTC m=+2341.375160675"
Sep 09 21:31:43.314678 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:43.314631    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:52.314569 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:52.314526    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:52.327249 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:52.327220    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:31:52.773026 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:31:52.772935    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:32:02.397917 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.397876    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"]
Sep 09 21:32:02.408001 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:32:02.407960    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:32:02.408001 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:32:02.408001 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:32:02.408001 ip-10-0-140-132 kubenswrapper[2583]:  > containerID="f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456" cmd=["/bin/sleep","15"]
Sep 09 21:32:02.408295 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.408020    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" message="" err=<
Sep 09 21:32:02.408295 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:32:02.408295 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:32:02.408295 ip-10-0-140-132 kubenswrapper[2583]:  >
Sep 09 21:32:02.408295 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:32:02.408033    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:32:02.408295 ip-10-0-140-132 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:32:02.408295 ip-10-0-140-132 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 09 21:32:02.408295 ip-10-0-140-132 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" podUID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerName="main" containerID="cri-o://f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456"
Sep 09 21:32:02.408295 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.408082    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" podUID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerName="main" containerID="cri-o://f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456" gracePeriod=60
Sep 09 21:32:02.661408 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.661300    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" podUID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 09 21:32:02.760137 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.760093    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:32:02.814010 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.813967    2583 generic.go:356] "Generic (PLEG): container finished" podID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerID="f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456" exitCode=0
Sep 09 21:32:02.814174 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.814035    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"
Sep 09 21:32:02.814174 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.814051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" event={"ID":"2734d600-0f4b-439d-9a61-18a3b91a14a1","Type":"ContainerDied","Data":"f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456"}
Sep 09 21:32:02.814174 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.814124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr" event={"ID":"2734d600-0f4b-439d-9a61-18a3b91a14a1","Type":"ContainerDied","Data":"d9ceed9e6c1b03baa8bc2ee1e0cde170f4c2b91938e68f74553bd7d6f7d02f63"}
Sep 09 21:32:02.814174 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.814138    2583 scope.go:117] "RemoveContainer" containerID="f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456"
Sep 09 21:32:02.824178 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.824151    2583 scope.go:117] "RemoveContainer" containerID="267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113"
Sep 09 21:32:02.833363 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.833339    2583 scope.go:117] "RemoveContainer" containerID="f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456"
Sep 09 21:32:02.833730 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:32:02.833711    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456\": container with ID starting with f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456 not found: ID does not exist" containerID="f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456"
Sep 09 21:32:02.833786 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.833739    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456"} err="failed to get container status \"f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456\": rpc error: code = NotFound desc = could not find container \"f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456\": container with ID starting with f3c222b6f70c9de581d3864f71e5c5727dccf6ec2e90806e9b60523cf1dc1456 not found: ID does not exist"
Sep 09 21:32:02.833786 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.833758    2583 scope.go:117] "RemoveContainer" containerID="267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113"
Sep 09 21:32:02.834028 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:32:02.834000    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113\": container with ID starting with 267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113 not found: ID does not exist" containerID="267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113"
Sep 09 21:32:02.834089 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.834037    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113"} err="failed to get container status \"267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113\": rpc error: code = NotFound desc = could not find container \"267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113\": container with ID starting with 267f47348042055b10ee86d5afc968b9d923a57fac160fcda44033a12aaca113 not found: ID does not exist"
Sep 09 21:32:02.874483 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874442    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbmk5\" (UniqueName: \"kubernetes.io/projected/2734d600-0f4b-439d-9a61-18a3b91a14a1-kube-api-access-wbmk5\") pod \"2734d600-0f4b-439d-9a61-18a3b91a14a1\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") "
Sep 09 21:32:02.874683 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-home\") pod \"2734d600-0f4b-439d-9a61-18a3b91a14a1\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") "
Sep 09 21:32:02.874683 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874518    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-tmp-dir\") pod \"2734d600-0f4b-439d-9a61-18a3b91a14a1\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") "
Sep 09 21:32:02.874683 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874544    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-model-cache\") pod \"2734d600-0f4b-439d-9a61-18a3b91a14a1\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") "
Sep 09 21:32:02.874683 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874576    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2734d600-0f4b-439d-9a61-18a3b91a14a1-cabundle-cert\") pod \"2734d600-0f4b-439d-9a61-18a3b91a14a1\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") "
Sep 09 21:32:02.874683 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874624    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-kserve-provision-location\") pod \"2734d600-0f4b-439d-9a61-18a3b91a14a1\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") "
Sep 09 21:32:02.874683 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874661    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2734d600-0f4b-439d-9a61-18a3b91a14a1-tls-certs\") pod \"2734d600-0f4b-439d-9a61-18a3b91a14a1\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") "
Sep 09 21:32:02.874997 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874694    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-dshm\") pod \"2734d600-0f4b-439d-9a61-18a3b91a14a1\" (UID: \"2734d600-0f4b-439d-9a61-18a3b91a14a1\") "
Sep 09 21:32:02.874997 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874858    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-home" (OuterVolumeSpecName: "home") pod "2734d600-0f4b-439d-9a61-18a3b91a14a1" (UID: "2734d600-0f4b-439d-9a61-18a3b91a14a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.874997 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874852    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "2734d600-0f4b-439d-9a61-18a3b91a14a1" (UID: "2734d600-0f4b-439d-9a61-18a3b91a14a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.874997 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.874958    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2734d600-0f4b-439d-9a61-18a3b91a14a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2734d600-0f4b-439d-9a61-18a3b91a14a1" (UID: "2734d600-0f4b-439d-9a61-18a3b91a14a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:32:02.875240 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.875082    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.875240 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.875134    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.875240 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.875153    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2734d600-0f4b-439d-9a61-18a3b91a14a1-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.875380 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.875235    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2734d600-0f4b-439d-9a61-18a3b91a14a1" (UID: "2734d600-0f4b-439d-9a61-18a3b91a14a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.877528 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.877500    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-dshm" (OuterVolumeSpecName: "dshm") pod "2734d600-0f4b-439d-9a61-18a3b91a14a1" (UID: "2734d600-0f4b-439d-9a61-18a3b91a14a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.877528 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.877512    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2734d600-0f4b-439d-9a61-18a3b91a14a1-kube-api-access-wbmk5" (OuterVolumeSpecName: "kube-api-access-wbmk5") pod "2734d600-0f4b-439d-9a61-18a3b91a14a1" (UID: "2734d600-0f4b-439d-9a61-18a3b91a14a1"). InnerVolumeSpecName "kube-api-access-wbmk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:32:02.877678 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.877524    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2734d600-0f4b-439d-9a61-18a3b91a14a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2734d600-0f4b-439d-9a61-18a3b91a14a1" (UID: "2734d600-0f4b-439d-9a61-18a3b91a14a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:32:02.897911 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.897862    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2734d600-0f4b-439d-9a61-18a3b91a14a1" (UID: "2734d600-0f4b-439d-9a61-18a3b91a14a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.976084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.975990    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbmk5\" (UniqueName: \"kubernetes.io/projected/2734d600-0f4b-439d-9a61-18a3b91a14a1-kube-api-access-wbmk5\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.976084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.976044    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.976084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.976058    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.976084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.976069    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2734d600-0f4b-439d-9a61-18a3b91a14a1-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.976084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:02.976077    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2734d600-0f4b-439d-9a61-18a3b91a14a1-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:03.144047 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:03.144003    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"]
Sep 09 21:32:03.153497 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:03.153463    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-lhbnr"]
Sep 09 21:32:03.865042 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:03.865009    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2734d600-0f4b-439d-9a61-18a3b91a14a1" path="/var/lib/kubelet/pods/2734d600-0f4b-439d-9a61-18a3b91a14a1/volumes"
Sep 09 21:32:20.341431 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:20.341395    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"]
Sep 09 21:32:27.680507 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.680469    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"]
Sep 09 21:32:27.680939 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.680921    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerName="storage-initializer"
Sep 09 21:32:27.681004 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.680941    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerName="storage-initializer"
Sep 09 21:32:27.681004 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.680955    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerName="main"
Sep 09 21:32:27.681004 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.680961    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerName="main"
Sep 09 21:32:27.681136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.681027    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2734d600-0f4b-439d-9a61-18a3b91a14a1" containerName="main"
Sep 09 21:32:27.683937 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.683919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.690234 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.690205    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 09 21:32:27.691832 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.691808    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 09 21:32:27.691832 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.691825    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 09 21:32:27.692009 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.691836    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-xlmvt\""
Sep 09 21:32:27.700684 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.700657    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"]
Sep 09 21:32:27.704183 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.704157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.704290 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.704200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-home\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.704290 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.704226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqcq\" (UniqueName: \"kubernetes.io/projected/b11e5c61-ef46-47a2-b2a3-36565122d505-kube-api-access-cxqcq\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.704290 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.704281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e5c61-ef46-47a2-b2a3-36565122d505-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.704466 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.704307    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.704466 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.704349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-dshm\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.805178 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.805130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e5c61-ef46-47a2-b2a3-36565122d505-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.805178 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.805185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.805459 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.805226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-dshm\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.805459 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.805299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.805459 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.805329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-home\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.805459 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.805354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqcq\" (UniqueName: \"kubernetes.io/projected/b11e5c61-ef46-47a2-b2a3-36565122d505-kube-api-access-cxqcq\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.805668 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.805643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.805738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.805717    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.805784 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.805759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-home\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.807733 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.807710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-dshm\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.807861 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.807844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e5c61-ef46-47a2-b2a3-36565122d505-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.815551 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.815519    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqcq\" (UniqueName: \"kubernetes.io/projected/b11e5c61-ef46-47a2-b2a3-36565122d505-kube-api-access-cxqcq\") pod \"delw-v1-kserve-6cf8cbb847-nw2r4\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:27.995380 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:27.995286    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:28.274835 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:28.274806    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"]
Sep 09 21:32:28.278162 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:32:28.278125    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11e5c61_ef46_47a2_b2a3_36565122d505.slice/crio-3964c350f7cb10a51e3a1ba24c9e6dedbdf25141f1f2ad8b9531b160de4ca309 WatchSource:0}: Error finding container 3964c350f7cb10a51e3a1ba24c9e6dedbdf25141f1f2ad8b9531b160de4ca309: Status 404 returned error can't find the container with id 3964c350f7cb10a51e3a1ba24c9e6dedbdf25141f1f2ad8b9531b160de4ca309
Sep 09 21:32:28.937747 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:28.937711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4" event={"ID":"b11e5c61-ef46-47a2-b2a3-36565122d505","Type":"ContainerStarted","Data":"34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346"}
Sep 09 21:32:28.938162 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:28.937764    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4" event={"ID":"b11e5c61-ef46-47a2-b2a3-36565122d505","Type":"ContainerStarted","Data":"3964c350f7cb10a51e3a1ba24c9e6dedbdf25141f1f2ad8b9531b160de4ca309"}
Sep 09 21:32:28.995785 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:28.995748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:35.361949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.361881    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6" podUID="c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" containerName="main" containerID="cri-o://bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428" gracePeriod=45
Sep 09 21:32:35.612030 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.611966    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:32:35.639752 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.639689    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4" podStartSLOduration=8.639672274 podStartE2EDuration="8.639672274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:32:28.96375734 +0000 UTC m=+2387.588231283" watchObservedRunningTime="2026-09-09 21:32:35.639672274 +0000 UTC m=+2394.264146215"
Sep 09 21:32:35.674655 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.674619    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tls-certs\") pod \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") "
Sep 09 21:32:35.674655 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.674658    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bwrs\" (UniqueName: \"kubernetes.io/projected/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-kube-api-access-9bwrs\") pod \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") "
Sep 09 21:32:35.674877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.674680    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-dshm\") pod \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") "
Sep 09 21:32:35.674877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.674716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-model-cache\") pod \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") "
Sep 09 21:32:35.674877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.674739    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tmp-dir\") pod \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") "
Sep 09 21:32:35.674877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.674754    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-home\") pod \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\" (UID: \"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f\") "
Sep 09 21:32:35.675068 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.675018    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" (UID: "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:35.675068 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.675031    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" (UID: "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:35.675238 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.675208    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-home" (OuterVolumeSpecName: "home") pod "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" (UID: "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:35.677317 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.677290    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" (UID: "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:32:35.677434 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.677322    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-dshm" (OuterVolumeSpecName: "dshm") pod "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" (UID: "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:35.677434 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.677333    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-kube-api-access-9bwrs" (OuterVolumeSpecName: "kube-api-access-9bwrs") pod "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" (UID: "c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f"). InnerVolumeSpecName "kube-api-access-9bwrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:32:35.776243 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.776206    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:35.776243 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.776236    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bwrs\" (UniqueName: \"kubernetes.io/projected/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-kube-api-access-9bwrs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:35.776243 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.776250    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:35.776476 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.776259    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:35.776476 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.776268    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:35.776476 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.776275    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:32:35.973247 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.973207    2583 generic.go:356] "Generic (PLEG): container finished" podID="c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" containerID="bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428" exitCode=0
Sep 09 21:32:35.973394 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.973258    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"
Sep 09 21:32:35.973394 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.973266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6" event={"ID":"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f","Type":"ContainerDied","Data":"bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428"}
Sep 09 21:32:35.973394 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.973330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6" event={"ID":"c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f","Type":"ContainerDied","Data":"358aa76f590d78e99c8a0d051b7c0b2e12f25bec72e172471fa5da51b2ab9959"}
Sep 09 21:32:35.973394 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.973373    2583 scope.go:117] "RemoveContainer" containerID="bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428"
Sep 09 21:32:35.982399 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.982377    2583 scope.go:117] "RemoveContainer" containerID="bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428"
Sep 09 21:32:35.982704 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:32:35.982685    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428\": container with ID starting with bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428 not found: ID does not exist" containerID="bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428"
Sep 09 21:32:35.982754 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.982715    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428"} err="failed to get container status \"bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428\": rpc error: code = NotFound desc = could not find container \"bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428\": container with ID starting with bf92b43662d68ff71485d3d4dc36d4b95a25bcd71b19bcf04110da0bc4f48428 not found: ID does not exist"
Sep 09 21:32:35.997765 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:35.997735    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"]
Sep 09 21:32:36.005169 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:36.005140    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-448l6"]
Sep 09 21:32:37.864870 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:37.864836    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" path="/var/lib/kubelet/pods/c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f/volumes"
Sep 09 21:32:37.995711 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:37.995662    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:38.007974 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:38.007940    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:38.998450 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:38.998420    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:32:39.884858 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:39.884826    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"]
Sep 09 21:32:39.885192 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:39.885176    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" containerName="main"
Sep 09 21:32:39.885251 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:39.885196    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" containerName="main"
Sep 09 21:32:39.885304 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:39.885291    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43d0ebc-4ae0-4c20-b8fa-9a5803f8737f" containerName="main"
Sep 09 21:32:39.890175 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:39.890152    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:39.892995 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:39.892970    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 09 21:32:39.901254 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:39.901226    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"]
Sep 09 21:32:40.015937 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.015900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sxrb\" (UniqueName: \"kubernetes.io/projected/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-kube-api-access-2sxrb\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.016379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.015944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.016379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.015971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.016379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.016033    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.016379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.016073    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-home\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.016379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.016146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.117007 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.116963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-home\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.117242 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.117018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.117242 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.117094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxrb\" (UniqueName: \"kubernetes.io/projected/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-kube-api-access-2sxrb\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.117242 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.117162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.117242 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.117191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.117461 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.117259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.117523 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.117468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-home\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.117523 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.117496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.117636 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.117602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.119616 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.119591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.120059 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.120040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.128806 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.128781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxrb\" (UniqueName: \"kubernetes.io/projected/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-kube-api-access-2sxrb\") pod \"delw-v2-kserve-5b84cc7ddf-8h98c\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.200832 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.200729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:40.484170 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:40.484140    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"]
Sep 09 21:32:40.486939 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:32:40.486914    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc140dcd9_2916_4a47_a475_1f18f7b6b4bc.slice/crio-3d44d5fe00d97305aec3c53446a28dd09aeb0fc3bb34990e82b1e29b12694df7 WatchSource:0}: Error finding container 3d44d5fe00d97305aec3c53446a28dd09aeb0fc3bb34990e82b1e29b12694df7: Status 404 returned error can't find the container with id 3d44d5fe00d97305aec3c53446a28dd09aeb0fc3bb34990e82b1e29b12694df7
Sep 09 21:32:41.000493 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:41.000457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c" event={"ID":"c140dcd9-2916-4a47-a475-1f18f7b6b4bc","Type":"ContainerStarted","Data":"b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654"}
Sep 09 21:32:41.000666 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:41.000512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c" event={"ID":"c140dcd9-2916-4a47-a475-1f18f7b6b4bc","Type":"ContainerStarted","Data":"3d44d5fe00d97305aec3c53446a28dd09aeb0fc3bb34990e82b1e29b12694df7"}
Sep 09 21:32:41.022204 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:41.022153    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c" podStartSLOduration=2.022137716 podStartE2EDuration="2.022137716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:32:41.020618885 +0000 UTC m=+2399.645092848" watchObservedRunningTime="2026-09-09 21:32:41.022137716 +0000 UTC m=+2399.646611654"
Sep 09 21:32:41.201548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:41.201509    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:50.201389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:50.201346    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:50.214215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:50.214189    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:32:51.059726 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:32:51.059693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:33:30.280457 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:30.280418    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"]
Sep 09 21:33:45.300516 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.300446    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c" podUID="c140dcd9-2916-4a47-a475-1f18f7b6b4bc" containerName="main" containerID="cri-o://b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654" gracePeriod=45
Sep 09 21:33:45.550952 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.550883    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:33:45.630660 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.630616    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-model-cache\") pod \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") "
Sep 09 21:33:45.630660 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.630662    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-dshm\") pod \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") "
Sep 09 21:33:45.630907 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.630700    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-home\") pod \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") "
Sep 09 21:33:45.630907 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.630721    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tmp-dir\") pod \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") "
Sep 09 21:33:45.630907 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.630758    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tls-certs\") pod \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") "
Sep 09 21:33:45.630907 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.630796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sxrb\" (UniqueName: \"kubernetes.io/projected/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-kube-api-access-2sxrb\") pod \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\" (UID: \"c140dcd9-2916-4a47-a475-1f18f7b6b4bc\") "
Sep 09 21:33:45.631136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.630943    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c140dcd9-2916-4a47-a475-1f18f7b6b4bc" (UID: "c140dcd9-2916-4a47-a475-1f18f7b6b4bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:33:45.631136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.631013    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-home" (OuterVolumeSpecName: "home") pod "c140dcd9-2916-4a47-a475-1f18f7b6b4bc" (UID: "c140dcd9-2916-4a47-a475-1f18f7b6b4bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:33:45.631136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.631073    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c140dcd9-2916-4a47-a475-1f18f7b6b4bc" (UID: "c140dcd9-2916-4a47-a475-1f18f7b6b4bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:33:45.631286 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.631196    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:33:45.631286 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.631215    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:33:45.631286 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.631232    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:33:45.633257 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.633227    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-kube-api-access-2sxrb" (OuterVolumeSpecName: "kube-api-access-2sxrb") pod "c140dcd9-2916-4a47-a475-1f18f7b6b4bc" (UID: "c140dcd9-2916-4a47-a475-1f18f7b6b4bc"). InnerVolumeSpecName "kube-api-access-2sxrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:33:45.633355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.633254    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c140dcd9-2916-4a47-a475-1f18f7b6b4bc" (UID: "c140dcd9-2916-4a47-a475-1f18f7b6b4bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:33:45.633355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.633265    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-dshm" (OuterVolumeSpecName: "dshm") pod "c140dcd9-2916-4a47-a475-1f18f7b6b4bc" (UID: "c140dcd9-2916-4a47-a475-1f18f7b6b4bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:33:45.731788 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.731756    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:33:45.731788 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.731784    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sxrb\" (UniqueName: \"kubernetes.io/projected/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-kube-api-access-2sxrb\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:33:45.731788 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:45.731793    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c140dcd9-2916-4a47-a475-1f18f7b6b4bc-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:33:46.314484 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.314449    2583 generic.go:356] "Generic (PLEG): container finished" podID="c140dcd9-2916-4a47-a475-1f18f7b6b4bc" containerID="b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654" exitCode=0
Sep 09 21:33:46.314887 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.314495    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"
Sep 09 21:33:46.314887 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.314525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c" event={"ID":"c140dcd9-2916-4a47-a475-1f18f7b6b4bc","Type":"ContainerDied","Data":"b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654"}
Sep 09 21:33:46.314887 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.314569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c" event={"ID":"c140dcd9-2916-4a47-a475-1f18f7b6b4bc","Type":"ContainerDied","Data":"3d44d5fe00d97305aec3c53446a28dd09aeb0fc3bb34990e82b1e29b12694df7"}
Sep 09 21:33:46.314887 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.314596    2583 scope.go:117] "RemoveContainer" containerID="b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654"
Sep 09 21:33:46.323723 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.323706    2583 scope.go:117] "RemoveContainer" containerID="b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654"
Sep 09 21:33:46.324018 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:33:46.323993    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654\": container with ID starting with b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654 not found: ID does not exist" containerID="b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654"
Sep 09 21:33:46.324153 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.324025    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654"} err="failed to get container status \"b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654\": rpc error: code = NotFound desc = could not find container \"b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654\": container with ID starting with b40f1ed3af49e21577026e34d309c90e940782f7b32c8b87b8b6a3f134c0a654 not found: ID does not exist"
Sep 09 21:33:46.347712 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.347673    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"]
Sep 09 21:33:46.350901 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.350868    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"]
Sep 09 21:33:46.364462 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:46.364426    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-8h98c"]
Sep 09 21:33:47.865266 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:47.865234    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c140dcd9-2916-4a47-a475-1f18f7b6b4bc" path="/var/lib/kubelet/pods/c140dcd9-2916-4a47-a475-1f18f7b6b4bc/volumes"
Sep 09 21:33:56.405604 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.405567    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"]
Sep 09 21:33:56.405987 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.405912    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c140dcd9-2916-4a47-a475-1f18f7b6b4bc" containerName="main"
Sep 09 21:33:56.405987 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.405923    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c140dcd9-2916-4a47-a475-1f18f7b6b4bc" containerName="main"
Sep 09 21:33:56.406062 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.405994    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c140dcd9-2916-4a47-a475-1f18f7b6b4bc" containerName="main"
Sep 09 21:33:56.409097 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.409078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.414838 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.414814    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 09 21:33:56.414983 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.414839    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 09 21:33:56.414983 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.414820    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 09 21:33:56.414983 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.414816    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-zdx7b\""
Sep 09 21:33:56.465686 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.465646    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"]
Sep 09 21:33:56.526945 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.526911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-home\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.526945 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.526947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.526945 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.526963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spsmz\" (UniqueName: \"kubernetes.io/projected/36253d26-396e-4e4d-b1f3-bcb557ff5563-kube-api-access-spsmz\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.527269 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.527021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-dshm\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.527269 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.527058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.527269 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.527074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36253d26-396e-4e4d-b1f3-bcb557ff5563-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.628135 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.628084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.628135 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.628140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36253d26-396e-4e4d-b1f3-bcb557ff5563-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.628400 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.628190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-home\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.628400 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.628210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.628400 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.628225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spsmz\" (UniqueName: \"kubernetes.io/projected/36253d26-396e-4e4d-b1f3-bcb557ff5563-kube-api-access-spsmz\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.628400 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.628274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-dshm\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.628607 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.628582    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.628643 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.628605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-home\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.628696 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.628673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.630902 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.630870    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36253d26-396e-4e4d-b1f3-bcb557ff5563-dshm\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.631136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.631097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36253d26-396e-4e4d-b1f3-bcb557ff5563-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.642332 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.642306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spsmz\" (UniqueName: \"kubernetes.io/projected/36253d26-396e-4e4d-b1f3-bcb557ff5563-kube-api-access-spsmz\") pod \"roll-v1-kserve-69cf8bcfc-rvtwz\" (UID: \"36253d26-396e-4e4d-b1f3-bcb557ff5563\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:56.720759 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:56.720665    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:33:57.002075 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:57.002046    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"]
Sep 09 21:33:57.003718 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:33:57.003686    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36253d26_396e_4e4d_b1f3_bcb557ff5563.slice/crio-7f0f6a0816c0b77bdb1d894999952c5e6395f34f2a31a660040982f0fba91268 WatchSource:0}: Error finding container 7f0f6a0816c0b77bdb1d894999952c5e6395f34f2a31a660040982f0fba91268: Status 404 returned error can't find the container with id 7f0f6a0816c0b77bdb1d894999952c5e6395f34f2a31a660040982f0fba91268
Sep 09 21:33:57.366519 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:57.366485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz" event={"ID":"36253d26-396e-4e4d-b1f3-bcb557ff5563","Type":"ContainerStarted","Data":"782f74feef6a0e7684055ed591d35f4355e5ae7ec61ac1f82a4bc30b87b3c1a3"}
Sep 09 21:33:57.366754 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:57.366546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz" event={"ID":"36253d26-396e-4e4d-b1f3-bcb557ff5563","Type":"ContainerStarted","Data":"7f0f6a0816c0b77bdb1d894999952c5e6395f34f2a31a660040982f0fba91268"}
Sep 09 21:33:57.390165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:57.390091    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz" podStartSLOduration=1.390072371 podStartE2EDuration="1.390072371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:57.388924508 +0000 UTC m=+2476.013398452" watchObservedRunningTime="2026-09-09 21:33:57.390072371 +0000 UTC m=+2476.014546311"
Sep 09 21:33:57.721271 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:33:57.721170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:34:01.373012 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.372945    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4" podUID="b11e5c61-ef46-47a2-b2a3-36565122d505" containerName="main" containerID="cri-o://34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346" gracePeriod=45
Sep 09 21:34:01.622907 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.622880    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:34:01.671012 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.670909    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-tmp-dir\") pod \"b11e5c61-ef46-47a2-b2a3-36565122d505\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") "
Sep 09 21:34:01.671012 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.670975    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-home\") pod \"b11e5c61-ef46-47a2-b2a3-36565122d505\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") "
Sep 09 21:34:01.671284 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.671023    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e5c61-ef46-47a2-b2a3-36565122d505-tls-certs\") pod \"b11e5c61-ef46-47a2-b2a3-36565122d505\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") "
Sep 09 21:34:01.671284 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.671064    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-dshm\") pod \"b11e5c61-ef46-47a2-b2a3-36565122d505\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") "
Sep 09 21:34:01.671284 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.671117    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxqcq\" (UniqueName: \"kubernetes.io/projected/b11e5c61-ef46-47a2-b2a3-36565122d505-kube-api-access-cxqcq\") pod \"b11e5c61-ef46-47a2-b2a3-36565122d505\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") "
Sep 09 21:34:01.671284 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.671188    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-model-cache\") pod \"b11e5c61-ef46-47a2-b2a3-36565122d505\" (UID: \"b11e5c61-ef46-47a2-b2a3-36565122d505\") "
Sep 09 21:34:01.671485 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.671410    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b11e5c61-ef46-47a2-b2a3-36565122d505" (UID: "b11e5c61-ef46-47a2-b2a3-36565122d505"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:34:01.671485 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.671432    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-home" (OuterVolumeSpecName: "home") pod "b11e5c61-ef46-47a2-b2a3-36565122d505" (UID: "b11e5c61-ef46-47a2-b2a3-36565122d505"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:34:01.671592 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.671562    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:34:01.671649 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.671619    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:34:01.671716 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.671649    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-model-cache" (OuterVolumeSpecName: "model-cache") pod "b11e5c61-ef46-47a2-b2a3-36565122d505" (UID: "b11e5c61-ef46-47a2-b2a3-36565122d505"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:34:01.673564 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.673536    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-dshm" (OuterVolumeSpecName: "dshm") pod "b11e5c61-ef46-47a2-b2a3-36565122d505" (UID: "b11e5c61-ef46-47a2-b2a3-36565122d505"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:34:01.673700 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.673604    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11e5c61-ef46-47a2-b2a3-36565122d505-kube-api-access-cxqcq" (OuterVolumeSpecName: "kube-api-access-cxqcq") pod "b11e5c61-ef46-47a2-b2a3-36565122d505" (UID: "b11e5c61-ef46-47a2-b2a3-36565122d505"). InnerVolumeSpecName "kube-api-access-cxqcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:34:01.673700 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.673641    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11e5c61-ef46-47a2-b2a3-36565122d505-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b11e5c61-ef46-47a2-b2a3-36565122d505" (UID: "b11e5c61-ef46-47a2-b2a3-36565122d505"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:34:01.772193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.772152    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e5c61-ef46-47a2-b2a3-36565122d505-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:34:01.772193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.772181    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:34:01.772193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.772190    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxqcq\" (UniqueName: \"kubernetes.io/projected/b11e5c61-ef46-47a2-b2a3-36565122d505-kube-api-access-cxqcq\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:34:01.772193 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:01.772199    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e5c61-ef46-47a2-b2a3-36565122d505-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:34:02.391408 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:02.391370    2583 generic.go:356] "Generic (PLEG): container finished" podID="b11e5c61-ef46-47a2-b2a3-36565122d505" containerID="34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346" exitCode=0
Sep 09 21:34:02.391794 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:02.391429    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"
Sep 09 21:34:02.391794 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:02.391458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4" event={"ID":"b11e5c61-ef46-47a2-b2a3-36565122d505","Type":"ContainerDied","Data":"34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346"}
Sep 09 21:34:02.391794 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:02.391519    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4" event={"ID":"b11e5c61-ef46-47a2-b2a3-36565122d505","Type":"ContainerDied","Data":"3964c350f7cb10a51e3a1ba24c9e6dedbdf25141f1f2ad8b9531b160de4ca309"}
Sep 09 21:34:02.391794 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:02.391542    2583 scope.go:117] "RemoveContainer" containerID="34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346"
Sep 09 21:34:02.400468 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:02.400447    2583 scope.go:117] "RemoveContainer" containerID="34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346"
Sep 09 21:34:02.400806 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:34:02.400784    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346\": container with ID starting with 34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346 not found: ID does not exist" containerID="34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346"
Sep 09 21:34:02.400890 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:02.400814    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346"} err="failed to get container status \"34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346\": rpc error: code = NotFound desc = could not find container \"34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346\": container with ID starting with 34825f75c3c044850e854e5ab335a8748c0d07790a897184a150f16405b39346 not found: ID does not exist"
Sep 09 21:34:02.412796 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:02.412756    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"]
Sep 09 21:34:02.417777 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:02.417749    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-nw2r4"]
Sep 09 21:34:03.865318 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.865287    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b11e5c61-ef46-47a2-b2a3-36565122d505" path="/var/lib/kubelet/pods/b11e5c61-ef46-47a2-b2a3-36565122d505/volumes"
Sep 09 21:34:03.896502 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.896462    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"]
Sep 09 21:34:03.896889 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.896871    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11e5c61-ef46-47a2-b2a3-36565122d505" containerName="main"
Sep 09 21:34:03.896976 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.896893    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11e5c61-ef46-47a2-b2a3-36565122d505" containerName="main"
Sep 09 21:34:03.897030 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.896993    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11e5c61-ef46-47a2-b2a3-36565122d505" containerName="main"
Sep 09 21:34:03.901283 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.901259    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:03.904065 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.904043    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 09 21:34:03.911947 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.911922    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"]
Sep 09 21:34:03.989648 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.989611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:03.989830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.989658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-home\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:03.989830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.989741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:03.989830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.989775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb857d95-6ce2-409e-abde-48fcbdff1920-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:03.989830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.989810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-dshm\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:03.989830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:03.989829    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngzj\" (UniqueName: \"kubernetes.io/projected/fb857d95-6ce2-409e-abde-48fcbdff1920-kube-api-access-rngzj\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.090829 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.090793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-dshm\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.090829 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.090829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngzj\" (UniqueName: \"kubernetes.io/projected/fb857d95-6ce2-409e-abde-48fcbdff1920-kube-api-access-rngzj\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.091056 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.090864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.091056 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.090889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-home\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.091056 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.091016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.091241 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.091065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb857d95-6ce2-409e-abde-48fcbdff1920-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.091327 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.091296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.091388 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.091366    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-home\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.091491 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.091470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.093249 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.093226    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb857d95-6ce2-409e-abde-48fcbdff1920-dshm\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.093699 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.093679    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb857d95-6ce2-409e-abde-48fcbdff1920-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.100939 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.100917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngzj\" (UniqueName: \"kubernetes.io/projected/fb857d95-6ce2-409e-abde-48fcbdff1920-kube-api-access-rngzj\") pod \"roll-v2-kserve-d4dbf9d67-bxwdq\" (UID: \"fb857d95-6ce2-409e-abde-48fcbdff1920\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.212215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.212097    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:04.495063 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:04.494999    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"]
Sep 09 21:34:04.497650 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:34:04.497609    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb857d95_6ce2_409e_abde_48fcbdff1920.slice/crio-0b4576c8ab08797a900904fdf13f7e34786485e4beaedac11d668ecb18941e23 WatchSource:0}: Error finding container 0b4576c8ab08797a900904fdf13f7e34786485e4beaedac11d668ecb18941e23: Status 404 returned error can't find the container with id 0b4576c8ab08797a900904fdf13f7e34786485e4beaedac11d668ecb18941e23
Sep 09 21:34:05.408850 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:05.408812    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq" event={"ID":"fb857d95-6ce2-409e-abde-48fcbdff1920","Type":"ContainerStarted","Data":"eabb901b640bf6da7190cd97f2509edb67b68ce4127967840f5f97fdd5998509"}
Sep 09 21:34:05.409269 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:05.408873    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq" event={"ID":"fb857d95-6ce2-409e-abde-48fcbdff1920","Type":"ContainerStarted","Data":"0b4576c8ab08797a900904fdf13f7e34786485e4beaedac11d668ecb18941e23"}
Sep 09 21:34:05.431571 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:05.431520    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq" podStartSLOduration=2.431502166 podStartE2EDuration="2.431502166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:05.430595633 +0000 UTC m=+2484.055069575" watchObservedRunningTime="2026-09-09 21:34:05.431502166 +0000 UTC m=+2484.055976107"
Sep 09 21:34:06.212287 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:06.212253    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:06.721744 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:06.721707    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:34:06.734434 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:06.734404    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:34:07.430947 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:07.430912    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rvtwz"
Sep 09 21:34:14.213369 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:14.213327    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:14.226191 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:14.226166    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:34:14.460925 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:34:14.460899    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-bxwdq"
Sep 09 21:35:23.008044 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:23.008002    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-rollback-b8fdb441_roll-v1-kserve-69cf8bcfc-rvtwz_36253d26-396e-4e4d-b1f3-bcb557ff5563/main/0.log"
Sep 09 21:35:24.808750 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:24.808715    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-rollback-b8fdb441_roll-v2-kserve-d4dbf9d67-bxwdq_fb857d95-6ce2-409e-abde-48fcbdff1920/main/0.log"
Sep 09 21:35:47.108677 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.108632    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"]
Sep 09 21:35:47.112292 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.112270    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.116935 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.116909    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 09 21:35:47.116935 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.116926    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 09 21:35:47.117153 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.116912    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-b8kmc\""
Sep 09 21:35:47.117153 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.117031    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 09 21:35:47.188449 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.188413    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"]
Sep 09 21:35:47.229197 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.229158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.229197 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.229197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x685k\" (UniqueName: \"kubernetes.io/projected/2a4b4719-83be-498c-b1a7-671b9ce5d889-kube-api-access-x685k\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.229421 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.229219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-home\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.229421 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.229284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4b4719-83be-498c-b1a7-671b9ce5d889-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.229421 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.229398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-dshm\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.229526 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.229426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-model-cache\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.330332 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.330290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-dshm\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.330332 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.330336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-model-cache\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.330577 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.330457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.330577 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.330491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x685k\" (UniqueName: \"kubernetes.io/projected/2a4b4719-83be-498c-b1a7-671b9ce5d889-kube-api-access-x685k\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.330577 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.330517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-home\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.330577 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.330539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4b4719-83be-498c-b1a7-671b9ce5d889-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.330833 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.330809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-model-cache\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.331027 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.330915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.331027 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.330968    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-home\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.332894 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.332875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-dshm\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.333199 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.333177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4b4719-83be-498c-b1a7-671b9ce5d889-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.352813 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.352787    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x685k\" (UniqueName: \"kubernetes.io/projected/2a4b4719-83be-498c-b1a7-671b9ce5d889-kube-api-access-x685k\") pod \"owner-v1-kserve-75579ff5bc-nm68b\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.422965 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.422864    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:47.704044 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.704018    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"]
Sep 09 21:35:47.705971 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:35:47.705931    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4b4719_83be_498c_b1a7_671b9ce5d889.slice/crio-6ba5e3e329e1382311c3323ca849fc5bc57caba0202d81ae18344690901924de WatchSource:0}: Error finding container 6ba5e3e329e1382311c3323ca849fc5bc57caba0202d81ae18344690901924de: Status 404 returned error can't find the container with id 6ba5e3e329e1382311c3323ca849fc5bc57caba0202d81ae18344690901924de
Sep 09 21:35:47.895662 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.895619    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b" event={"ID":"2a4b4719-83be-498c-b1a7-671b9ce5d889","Type":"ContainerStarted","Data":"4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5"}
Sep 09 21:35:47.895875 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.895684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b" event={"ID":"2a4b4719-83be-498c-b1a7-671b9ce5d889","Type":"ContainerStarted","Data":"6ba5e3e329e1382311c3323ca849fc5bc57caba0202d81ae18344690901924de"}
Sep 09 21:35:47.927937 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:47.927860    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b" podStartSLOduration=0.92784013 podStartE2EDuration="927.84013ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:35:47.927270902 +0000 UTC m=+2586.551744855" watchObservedRunningTime="2026-09-09 21:35:47.92784013 +0000 UTC m=+2586.552314071"
Sep 09 21:35:48.423846 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:48.423808    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:57.423882 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:57.423847    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:57.436994 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:57.436964    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:35:57.951153 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:35:57.951125    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:36:03.897726 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.897647    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"]
Sep 09 21:36:03.901127 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.901078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:03.904898 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.904874    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 09 21:36:03.918634 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.918607    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"]
Sep 09 21:36:03.983877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.983836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:03.983877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.983873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-dshm\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:03.984189 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.983891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-home\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:03.984189 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.983934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-model-cache\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:03.984189 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.984055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:03.984189 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:03.984151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnjqv\" (UniqueName: \"kubernetes.io/projected/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-kube-api-access-dnjqv\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.084869 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.084834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.084869 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.084877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-dshm\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.085136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.084896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-home\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.085136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.084914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-model-cache\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.085136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.084999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.085136 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.085048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnjqv\" (UniqueName: \"kubernetes.io/projected/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-kube-api-access-dnjqv\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.085402 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.085360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-home\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.085552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.085409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-model-cache\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.085552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.085441    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.087225 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.087207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-dshm\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.087716 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.087695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.096755 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.096729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnjqv\" (UniqueName: \"kubernetes.io/projected/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-kube-api-access-dnjqv\") pod \"owner-v2-kserve-5cf8959f95-xsfvk\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.211696 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.211603    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:04.503703 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.503672    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"]
Sep 09 21:36:04.506441 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:36:04.506412    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495d0fbb_3e4f_4e06_8238_b9e67433f6b0.slice/crio-7363f0162170303574ecfdd33c0f215b90154ad8915934acd8ce6c188a5f7cfa WatchSource:0}: Error finding container 7363f0162170303574ecfdd33c0f215b90154ad8915934acd8ce6c188a5f7cfa: Status 404 returned error can't find the container with id 7363f0162170303574ecfdd33c0f215b90154ad8915934acd8ce6c188a5f7cfa
Sep 09 21:36:04.974742 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.974710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk" event={"ID":"495d0fbb-3e4f-4e06-8238-b9e67433f6b0","Type":"ContainerStarted","Data":"b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b"}
Sep 09 21:36:04.975141 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:04.974759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk" event={"ID":"495d0fbb-3e4f-4e06-8238-b9e67433f6b0","Type":"ContainerStarted","Data":"7363f0162170303574ecfdd33c0f215b90154ad8915934acd8ce6c188a5f7cfa"}
Sep 09 21:36:05.002785 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:05.002727    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk" podStartSLOduration=2.002708568 podStartE2EDuration="2.002708568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:36:05.001350972 +0000 UTC m=+2603.625824913" watchObservedRunningTime="2026-09-09 21:36:05.002708568 +0000 UTC m=+2603.627182509"
Sep 09 21:36:05.212363 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:05.212317    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:14.211959 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:14.211904    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:14.226768 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:14.226740    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:15.039826 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:15.039795    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:36:42.179317 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:42.179281    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"]
Sep 09 21:36:57.201141 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.201077    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b" podUID="2a4b4719-83be-498c-b1a7-671b9ce5d889" containerName="main" containerID="cri-o://4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5" gracePeriod=45
Sep 09 21:36:57.442690 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.442662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:36:57.584414 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.584382    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-model-cache\") pod \"2a4b4719-83be-498c-b1a7-671b9ce5d889\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") "
Sep 09 21:36:57.584592 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.584424    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-dshm\") pod \"2a4b4719-83be-498c-b1a7-671b9ce5d889\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") "
Sep 09 21:36:57.584592 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.584449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x685k\" (UniqueName: \"kubernetes.io/projected/2a4b4719-83be-498c-b1a7-671b9ce5d889-kube-api-access-x685k\") pod \"2a4b4719-83be-498c-b1a7-671b9ce5d889\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") "
Sep 09 21:36:57.584592 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.584479    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-tmp-dir\") pod \"2a4b4719-83be-498c-b1a7-671b9ce5d889\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") "
Sep 09 21:36:57.584592 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.584551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4b4719-83be-498c-b1a7-671b9ce5d889-tls-certs\") pod \"2a4b4719-83be-498c-b1a7-671b9ce5d889\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") "
Sep 09 21:36:57.584592 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.584581    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-home\") pod \"2a4b4719-83be-498c-b1a7-671b9ce5d889\" (UID: \"2a4b4719-83be-498c-b1a7-671b9ce5d889\") "
Sep 09 21:36:57.584861 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.584737    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a4b4719-83be-498c-b1a7-671b9ce5d889" (UID: "2a4b4719-83be-498c-b1a7-671b9ce5d889"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:57.584861 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.584787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a4b4719-83be-498c-b1a7-671b9ce5d889" (UID: "2a4b4719-83be-498c-b1a7-671b9ce5d889"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:57.584975 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.584883    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-home" (OuterVolumeSpecName: "home") pod "2a4b4719-83be-498c-b1a7-671b9ce5d889" (UID: "2a4b4719-83be-498c-b1a7-671b9ce5d889"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:57.585033 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.585012    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.585077 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.585037    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.585077 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.585045    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.586926 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.586893    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-dshm" (OuterVolumeSpecName: "dshm") pod "2a4b4719-83be-498c-b1a7-671b9ce5d889" (UID: "2a4b4719-83be-498c-b1a7-671b9ce5d889"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:57.587061 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.586966    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4b4719-83be-498c-b1a7-671b9ce5d889-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a4b4719-83be-498c-b1a7-671b9ce5d889" (UID: "2a4b4719-83be-498c-b1a7-671b9ce5d889"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:36:57.587061 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.586983    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4b4719-83be-498c-b1a7-671b9ce5d889-kube-api-access-x685k" (OuterVolumeSpecName: "kube-api-access-x685k") pod "2a4b4719-83be-498c-b1a7-671b9ce5d889" (UID: "2a4b4719-83be-498c-b1a7-671b9ce5d889"). InnerVolumeSpecName "kube-api-access-x685k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:36:57.686221 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.686173    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4b4719-83be-498c-b1a7-671b9ce5d889-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.686221 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.686214    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x685k\" (UniqueName: \"kubernetes.io/projected/2a4b4719-83be-498c-b1a7-671b9ce5d889-kube-api-access-x685k\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:36:57.686221 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:57.686225    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4b4719-83be-498c-b1a7-671b9ce5d889-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:36:58.242038 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:58.241938    2583 generic.go:356] "Generic (PLEG): container finished" podID="2a4b4719-83be-498c-b1a7-671b9ce5d889" containerID="4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5" exitCode=0
Sep 09 21:36:58.242038 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:58.241990    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"
Sep 09 21:36:58.242038 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:58.241995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b" event={"ID":"2a4b4719-83be-498c-b1a7-671b9ce5d889","Type":"ContainerDied","Data":"4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5"}
Sep 09 21:36:58.242614 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:58.242052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b" event={"ID":"2a4b4719-83be-498c-b1a7-671b9ce5d889","Type":"ContainerDied","Data":"6ba5e3e329e1382311c3323ca849fc5bc57caba0202d81ae18344690901924de"}
Sep 09 21:36:58.242614 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:58.242080    2583 scope.go:117] "RemoveContainer" containerID="4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5"
Sep 09 21:36:58.250940 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:58.250922    2583 scope.go:117] "RemoveContainer" containerID="4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5"
Sep 09 21:36:58.251304 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:36:58.251276    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5\": container with ID starting with 4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5 not found: ID does not exist" containerID="4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5"
Sep 09 21:36:58.251404 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:58.251316    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5"} err="failed to get container status \"4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5\": rpc error: code = NotFound desc = could not find container \"4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5\": container with ID starting with 4040e1d9afd773e23396ac2dc9fede9e4860403054ad7d9c265db17f2b63f0b5 not found: ID does not exist"
Sep 09 21:36:58.265702 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:58.265666    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"]
Sep 09 21:36:58.270934 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:58.270908    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-nm68b"]
Sep 09 21:36:59.865458 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:36:59.865423    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a4b4719-83be-498c-b1a7-671b9ce5d889" path="/var/lib/kubelet/pods/2a4b4719-83be-498c-b1a7-671b9ce5d889/volumes"
Sep 09 21:37:04.324678 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.324647    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"]
Sep 09 21:37:04.325096 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.325018    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4b4719-83be-498c-b1a7-671b9ce5d889" containerName="main"
Sep 09 21:37:04.325096 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.325031    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4b4719-83be-498c-b1a7-671b9ce5d889" containerName="main"
Sep 09 21:37:04.325096 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.325098    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4b4719-83be-498c-b1a7-671b9ce5d889" containerName="main"
Sep 09 21:37:04.329406 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.329387    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"]
Sep 09 21:37:04.329552 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.329533    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.335679 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.335648    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 09 21:37:04.335885 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.335761    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:37:04.335885 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.335809    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 09 21:37:04.337471 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.337314    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-vv2ml\""
Sep 09 21:37:04.340357 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.340334    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"]
Sep 09 21:37:04.451002 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.450959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.451002 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.451008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.451307 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.451027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.451307 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.451053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.451307 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.451076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428fbf26-5518-47d6-86eb-29a0d6c459ab-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.451307 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.451100    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.451307 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.451135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.451307 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.451278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8dzj\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-kube-api-access-b8dzj\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.451548 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.451336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552369 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8dzj\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-kube-api-access-b8dzj\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552568 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552568 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552568 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552568 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552568 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552801 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428fbf26-5518-47d6-86eb-29a0d6c459ab-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552801 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552801 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552961 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552887    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.552961 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.552921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.553067 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.553010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.553133 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.553076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.553332 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.553311    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428fbf26-5518-47d6-86eb-29a0d6c459ab-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.555176 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.555144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.555334 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.555318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.565732 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.565704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.568831 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.568809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8dzj\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-kube-api-access-b8dzj\") pod \"router-gateway-1-openshift-default-55f8b478f9-mtng5\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.641540 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.641508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:04.924499 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.924417    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"]
Sep 09 21:37:04.927532 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:37:04.927501    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428fbf26_5518_47d6_86eb_29a0d6c459ab.slice/crio-6e4501890f1092ce45558e9e93fd20b41a27254aef2bd2942252bca7a9e1c47e WatchSource:0}: Error finding container 6e4501890f1092ce45558e9e93fd20b41a27254aef2bd2942252bca7a9e1c47e: Status 404 returned error can't find the container with id 6e4501890f1092ce45558e9e93fd20b41a27254aef2bd2942252bca7a9e1c47e
Sep 09 21:37:04.929746 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.929714    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:37:04.929831 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.929778    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:37:04.929831 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:04.929805    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:37:05.292821 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:05.292783    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" event={"ID":"428fbf26-5518-47d6-86eb-29a0d6c459ab","Type":"ContainerStarted","Data":"7bf1d1d722a393416ec24f35d427f2bcb62a37c917617d557dea0002e136c036"}
Sep 09 21:37:05.293017 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:05.292885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" event={"ID":"428fbf26-5518-47d6-86eb-29a0d6c459ab","Type":"ContainerStarted","Data":"6e4501890f1092ce45558e9e93fd20b41a27254aef2bd2942252bca7a9e1c47e"}
Sep 09 21:37:05.642148 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:05.642028    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:05.647358 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:05.647330    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:05.674008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:05.673951    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" podStartSLOduration=1.6739344699999998 podStartE2EDuration="1.67393447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:37:05.318796394 +0000 UTC m=+2663.943270335" watchObservedRunningTime="2026-09-09 21:37:05.67393447 +0000 UTC m=+2664.298408411"
Sep 09 21:37:06.297631 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:06.297600    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:06.298949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:06.298927    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:19.350801 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.350758    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk" podUID="495d0fbb-3e4f-4e06-8238-b9e67433f6b0" containerName="main" containerID="cri-o://b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b" gracePeriod=45
Sep 09 21:37:19.593044 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.593017    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:37:19.680804 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.680706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-home\") pod \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") "
Sep 09 21:37:19.680804 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.680766    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-model-cache\") pod \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") "
Sep 09 21:37:19.681041 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.680845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tmp-dir\") pod \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") "
Sep 09 21:37:19.681041 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.680876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnjqv\" (UniqueName: \"kubernetes.io/projected/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-kube-api-access-dnjqv\") pod \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") "
Sep 09 21:37:19.681041 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.680912    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tls-certs\") pod \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") "
Sep 09 21:37:19.681041 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.680941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-dshm\") pod \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\" (UID: \"495d0fbb-3e4f-4e06-8238-b9e67433f6b0\") "
Sep 09 21:37:19.681041 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.681005    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "495d0fbb-3e4f-4e06-8238-b9e67433f6b0" (UID: "495d0fbb-3e4f-4e06-8238-b9e67433f6b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:19.681041 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.681031    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-home" (OuterVolumeSpecName: "home") pod "495d0fbb-3e4f-4e06-8238-b9e67433f6b0" (UID: "495d0fbb-3e4f-4e06-8238-b9e67433f6b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:19.681388 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.681196    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "495d0fbb-3e4f-4e06-8238-b9e67433f6b0" (UID: "495d0fbb-3e4f-4e06-8238-b9e67433f6b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:19.681388 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.681282    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:37:19.681388 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.681296    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:37:19.681388 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.681306    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:37:19.683287 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.683265    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-dshm" (OuterVolumeSpecName: "dshm") pod "495d0fbb-3e4f-4e06-8238-b9e67433f6b0" (UID: "495d0fbb-3e4f-4e06-8238-b9e67433f6b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:19.683398 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.683297    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-kube-api-access-dnjqv" (OuterVolumeSpecName: "kube-api-access-dnjqv") pod "495d0fbb-3e4f-4e06-8238-b9e67433f6b0" (UID: "495d0fbb-3e4f-4e06-8238-b9e67433f6b0"). InnerVolumeSpecName "kube-api-access-dnjqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:37:19.683398 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.683356    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "495d0fbb-3e4f-4e06-8238-b9e67433f6b0" (UID: "495d0fbb-3e4f-4e06-8238-b9e67433f6b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:37:19.781825 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.781783    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnjqv\" (UniqueName: \"kubernetes.io/projected/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-kube-api-access-dnjqv\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:37:19.781825 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.781813    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:37:19.781825 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:19.781827    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495d0fbb-3e4f-4e06-8238-b9e67433f6b0-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:37:20.367652 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:20.367617    2583 generic.go:356] "Generic (PLEG): container finished" podID="495d0fbb-3e4f-4e06-8238-b9e67433f6b0" containerID="b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b" exitCode=0
Sep 09 21:37:20.368071 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:20.367660    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"
Sep 09 21:37:20.368071 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:20.367693    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk" event={"ID":"495d0fbb-3e4f-4e06-8238-b9e67433f6b0","Type":"ContainerDied","Data":"b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b"}
Sep 09 21:37:20.368071 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:20.367765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk" event={"ID":"495d0fbb-3e4f-4e06-8238-b9e67433f6b0","Type":"ContainerDied","Data":"7363f0162170303574ecfdd33c0f215b90154ad8915934acd8ce6c188a5f7cfa"}
Sep 09 21:37:20.368071 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:20.367787    2583 scope.go:117] "RemoveContainer" containerID="b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b"
Sep 09 21:37:20.376576 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:20.376557    2583 scope.go:117] "RemoveContainer" containerID="b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b"
Sep 09 21:37:20.376850 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:37:20.376833    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b\": container with ID starting with b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b not found: ID does not exist" containerID="b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b"
Sep 09 21:37:20.376902 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:20.376857    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b"} err="failed to get container status \"b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b\": rpc error: code = NotFound desc = could not find container \"b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b\": container with ID starting with b1865d810a76bae477c896453d0314b63d7edaba057c542ca9a7a74817f5ca2b not found: ID does not exist"
Sep 09 21:37:20.391331 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:20.391298    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"]
Sep 09 21:37:20.398877 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:20.398851    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-xsfvk"]
Sep 09 21:37:21.873032 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:21.872995    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="495d0fbb-3e4f-4e06-8238-b9e67433f6b0" path="/var/lib/kubelet/pods/495d0fbb-3e4f-4e06-8238-b9e67433f6b0/volumes"
Sep 09 21:37:29.417409 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:29.417373    2583 generic.go:356] "Generic (PLEG): container finished" podID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerID="7bf1d1d722a393416ec24f35d427f2bcb62a37c917617d557dea0002e136c036" exitCode=0
Sep 09 21:37:29.417833 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:29.417446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" event={"ID":"428fbf26-5518-47d6-86eb-29a0d6c459ab","Type":"ContainerDied","Data":"7bf1d1d722a393416ec24f35d427f2bcb62a37c917617d557dea0002e136c036"}
Sep 09 21:37:29.417884 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:29.417852    2583 scope.go:117] "RemoveContainer" containerID="7bf1d1d722a393416ec24f35d427f2bcb62a37c917617d557dea0002e136c036"
Sep 09 21:37:29.418932 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:29.418897    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:37:29.419040 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:29.419018    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:37:29.419090 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:29.419075    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:37:30.427039 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:30.426992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" event={"ID":"428fbf26-5518-47d6-86eb-29a0d6c459ab","Type":"ContainerStarted","Data":"4c174e3958162c897ae886f12fa1c2daf51ccd4adf4184be47e849eb500c5d49"}
Sep 09 21:37:30.642461 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:30.642417    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:30.642688 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:30.642655    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 09 21:37:31.642757 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:31.642711    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 09 21:37:32.641937 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:32.641840    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 09 21:37:33.642782 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:33.642729    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 09 21:37:34.642931 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:34.642887    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 09 21:37:35.646723 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:35.646693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:35.647146 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:35.646966    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:37:35.647617 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:37:35.647600    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:39:44.948057 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:44.948025    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"]
Sep 09 21:39:44.948584 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:44.948283    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy" containerID="cri-o://4c174e3958162c897ae886f12fa1c2daf51ccd4adf4184be47e849eb500c5d49" gracePeriod=30
Sep 09 21:39:50.104139 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.104076    2583 generic.go:356] "Generic (PLEG): container finished" podID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerID="4c174e3958162c897ae886f12fa1c2daf51ccd4adf4184be47e849eb500c5d49" exitCode=0
Sep 09 21:39:50.104139 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.104140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" event={"ID":"428fbf26-5518-47d6-86eb-29a0d6c459ab","Type":"ContainerDied","Data":"4c174e3958162c897ae886f12fa1c2daf51ccd4adf4184be47e849eb500c5d49"}
Sep 09 21:39:50.104674 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.104206    2583 scope.go:117] "RemoveContainer" containerID="7bf1d1d722a393416ec24f35d427f2bcb62a37c917617d557dea0002e136c036"
Sep 09 21:39:50.207995 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.207966    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:39:50.279293 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279193    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-socket\") pod \"428fbf26-5518-47d6-86eb-29a0d6c459ab\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") "
Sep 09 21:39:50.279293 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-certs\") pod \"428fbf26-5518-47d6-86eb-29a0d6c459ab\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") "
Sep 09 21:39:50.279293 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279288    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-credential-socket\") pod \"428fbf26-5518-47d6-86eb-29a0d6c459ab\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") "
Sep 09 21:39:50.279593 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279318    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428fbf26-5518-47d6-86eb-29a0d6c459ab-istiod-ca-cert\") pod \"428fbf26-5518-47d6-86eb-29a0d6c459ab\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") "
Sep 09 21:39:50.279593 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279369    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-token\") pod \"428fbf26-5518-47d6-86eb-29a0d6c459ab\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") "
Sep 09 21:39:50.279593 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279392    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-data\") pod \"428fbf26-5518-47d6-86eb-29a0d6c459ab\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") "
Sep 09 21:39:50.279593 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279500    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "428fbf26-5518-47d6-86eb-29a0d6c459ab" (UID: "428fbf26-5518-47d6-86eb-29a0d6c459ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:50.279593 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279510    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "428fbf26-5518-47d6-86eb-29a0d6c459ab" (UID: "428fbf26-5518-47d6-86eb-29a0d6c459ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:50.279593 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279560    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-envoy\") pod \"428fbf26-5518-47d6-86eb-29a0d6c459ab\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") "
Sep 09 21:39:50.279909 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "428fbf26-5518-47d6-86eb-29a0d6c459ab" (UID: "428fbf26-5518-47d6-86eb-29a0d6c459ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:50.279909 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-podinfo\") pod \"428fbf26-5518-47d6-86eb-29a0d6c459ab\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") "
Sep 09 21:39:50.279909 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279702    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8dzj\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-kube-api-access-b8dzj\") pod \"428fbf26-5518-47d6-86eb-29a0d6c459ab\" (UID: \"428fbf26-5518-47d6-86eb-29a0d6c459ab\") "
Sep 09 21:39:50.279909 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.279778    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428fbf26-5518-47d6-86eb-29a0d6c459ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "428fbf26-5518-47d6-86eb-29a0d6c459ab" (UID: "428fbf26-5518-47d6-86eb-29a0d6c459ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:39:50.280213 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.280024    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-data\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:39:50.280213 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.280041    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-socket\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:39:50.280213 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.280056    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-workload-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:39:50.280213 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.280069    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428fbf26-5518-47d6-86eb-29a0d6c459ab-istiod-ca-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:39:50.280480 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.280453    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "428fbf26-5518-47d6-86eb-29a0d6c459ab" (UID: "428fbf26-5518-47d6-86eb-29a0d6c459ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:50.282796 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.282625    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "428fbf26-5518-47d6-86eb-29a0d6c459ab" (UID: "428fbf26-5518-47d6-86eb-29a0d6c459ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:39:50.282796 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.282650    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "428fbf26-5518-47d6-86eb-29a0d6c459ab" (UID: "428fbf26-5518-47d6-86eb-29a0d6c459ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:39:50.282961 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.282843    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "428fbf26-5518-47d6-86eb-29a0d6c459ab" (UID: "428fbf26-5518-47d6-86eb-29a0d6c459ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:50.283070 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.283045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-kube-api-access-b8dzj" (OuterVolumeSpecName: "kube-api-access-b8dzj") pod "428fbf26-5518-47d6-86eb-29a0d6c459ab" (UID: "428fbf26-5518-47d6-86eb-29a0d6c459ab"). InnerVolumeSpecName "kube-api-access-b8dzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:39:50.381389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.381355    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-credential-socket\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:39:50.381389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.381385    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-token\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:39:50.381389 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.381395    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-envoy\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:39:50.381627 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.381402    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428fbf26-5518-47d6-86eb-29a0d6c459ab-istio-podinfo\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:39:50.381627 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:50.381410    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8dzj\" (UniqueName: \"kubernetes.io/projected/428fbf26-5518-47d6-86eb-29a0d6c459ab-kube-api-access-b8dzj\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:39:51.110161 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:51.110128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5" event={"ID":"428fbf26-5518-47d6-86eb-29a0d6c459ab","Type":"ContainerDied","Data":"6e4501890f1092ce45558e9e93fd20b41a27254aef2bd2942252bca7a9e1c47e"}
Sep 09 21:39:51.110586 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:51.110188    2583 scope.go:117] "RemoveContainer" containerID="4c174e3958162c897ae886f12fa1c2daf51ccd4adf4184be47e849eb500c5d49"
Sep 09 21:39:51.110586 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:51.110136    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"
Sep 09 21:39:51.152786 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:51.152748    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"]
Sep 09 21:39:51.156446 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:51.156416    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-mtng5"]
Sep 09 21:39:51.865799 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:51.865760    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" path="/var/lib/kubelet/pods/428fbf26-5518-47d6-86eb-29a0d6c459ab/volumes"
Sep 09 21:39:56.704212 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.704158    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"]
Sep 09 21:39:56.707165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.705293    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy"
Sep 09 21:39:56.707165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.705327    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy"
Sep 09 21:39:56.707165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.705379    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495d0fbb-3e4f-4e06-8238-b9e67433f6b0" containerName="main"
Sep 09 21:39:56.707165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.705389    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="495d0fbb-3e4f-4e06-8238-b9e67433f6b0" containerName="main"
Sep 09 21:39:56.707165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.705400    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy"
Sep 09 21:39:56.707165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.705409    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy"
Sep 09 21:39:56.707165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.705566    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy"
Sep 09 21:39:56.707165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.705578    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="495d0fbb-3e4f-4e06-8238-b9e67433f6b0" containerName="main"
Sep 09 21:39:56.707165 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.705591    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="428fbf26-5518-47d6-86eb-29a0d6c459ab" containerName="istio-proxy"
Sep 09 21:39:56.712075 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.712046    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.715253 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.715227    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 09 21:39:56.715420 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.715383    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 09 21:39:56.715505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.715484    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:39:56.715822 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.715804    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 09 21:39:56.715900 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.715829    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-6s7bj\""
Sep 09 21:39:56.715968 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.715934    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 09 21:39:56.726314 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.726282    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"]
Sep 09 21:39:56.839672 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.839636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwqtp\" (UniqueName: \"kubernetes.io/projected/393b0a04-fb77-4799-85ab-456c19c86f11-kube-api-access-mwqtp\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.839860 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.839685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.839860 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.839742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.839860 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.839858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.840025 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.839879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.840025 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.839916    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393b0a04-fb77-4799-85ab-456c19c86f11-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.840025 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.839932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.840025 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.839961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/393b0a04-fb77-4799-85ab-456c19c86f11-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.940864 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.940823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.940864 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.940863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941173 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.940893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393b0a04-fb77-4799-85ab-456c19c86f11-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941173 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.941023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941173 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.941055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/393b0a04-fb77-4799-85ab-456c19c86f11-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941173 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.941089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqtp\" (UniqueName: \"kubernetes.io/projected/393b0a04-fb77-4799-85ab-456c19c86f11-kube-api-access-mwqtp\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.941179    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.941207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.941399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941562 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.941411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941562 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.941508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.941843 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.941710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.942053 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.942019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/393b0a04-fb77-4799-85ab-456c19c86f11-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.943416 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.943390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.943818 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.943796    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393b0a04-fb77-4799-85ab-456c19c86f11-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:56.956086 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:56.956003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqtp\" (UniqueName: \"kubernetes.io/projected/393b0a04-fb77-4799-85ab-456c19c86f11-kube-api-access-mwqtp\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:57.022387 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:57.022348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:39:57.326291 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:39:57.326250    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393b0a04_fb77_4799_85ab_456c19c86f11.slice/crio-57d48997f3de316ae3bddfc19ad7fb8e2ff9b3c824b7ad31137188e1b5c72ec8 WatchSource:0}: Error finding container 57d48997f3de316ae3bddfc19ad7fb8e2ff9b3c824b7ad31137188e1b5c72ec8: Status 404 returned error can't find the container with id 57d48997f3de316ae3bddfc19ad7fb8e2ff9b3c824b7ad31137188e1b5c72ec8
Sep 09 21:39:57.330474 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:57.330447    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"]
Sep 09 21:39:58.148633 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:58.148597    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" event={"ID":"393b0a04-fb77-4799-85ab-456c19c86f11","Type":"ContainerStarted","Data":"64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186"}
Sep 09 21:39:58.149052 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:39:58.148651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" event={"ID":"393b0a04-fb77-4799-85ab-456c19c86f11","Type":"ContainerStarted","Data":"57d48997f3de316ae3bddfc19ad7fb8e2ff9b3c824b7ad31137188e1b5c72ec8"}
Sep 09 21:40:00.159887 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:00.159853    2583 generic.go:356] "Generic (PLEG): container finished" podID="393b0a04-fb77-4799-85ab-456c19c86f11" containerID="64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186" exitCode=0
Sep 09 21:40:00.160293 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:00.159923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" event={"ID":"393b0a04-fb77-4799-85ab-456c19c86f11","Type":"ContainerDied","Data":"64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186"}
Sep 09 21:40:01.166669 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:01.166629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" event={"ID":"393b0a04-fb77-4799-85ab-456c19c86f11","Type":"ContainerStarted","Data":"734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87"}
Sep 09 21:40:01.202641 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:01.202588    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podStartSLOduration=2.909771663 podStartE2EDuration="5.202571253s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.29279959s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:40:01.201946894 +0000 UTC m=+2839.826420837" watchObservedRunningTime="2026-09-09 21:40:01.202571253 +0000 UTC m=+2839.827045215"
Sep 09 21:40:02.023030 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:02.022988    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:40:07.023116 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:07.023063    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:40:07.024195 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:07.024167    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:40:17.023360 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:17.023318    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:40:27.023611 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:27.023571    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:40:37.023772 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:37.023665    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:40:47.023539 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:47.023487    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:40:57.023447 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:40:57.023403    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:41:07.022921 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:41:07.022869    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:41:17.023737 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:41:17.023684    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:41:27.023367 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:41:27.023328    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:41:37.023125 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:41:37.023063    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:41:47.022869 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:41:47.022823    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:41:57.032982 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:41:57.032949    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:41:57.040951 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:41:57.040923    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:42:15.396910 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:15.396814    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"]
Sep 09 21:42:30.424795 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:30.424726    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" containerID="cri-o://734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87" gracePeriod=45
Sep 09 21:42:31.034384 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.034343    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 21:42:31.180449 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.180424    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:42:31.294217 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294175    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-home\") pod \"393b0a04-fb77-4799-85ab-456c19c86f11\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") "
Sep 09 21:42:31.294398 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294252    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwqtp\" (UniqueName: \"kubernetes.io/projected/393b0a04-fb77-4799-85ab-456c19c86f11-kube-api-access-mwqtp\") pod \"393b0a04-fb77-4799-85ab-456c19c86f11\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") "
Sep 09 21:42:31.294398 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294269    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393b0a04-fb77-4799-85ab-456c19c86f11-tls-certs\") pod \"393b0a04-fb77-4799-85ab-456c19c86f11\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") "
Sep 09 21:42:31.294398 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294289    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-dshm\") pod \"393b0a04-fb77-4799-85ab-456c19c86f11\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") "
Sep 09 21:42:31.294398 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294322    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-tmp-dir\") pod \"393b0a04-fb77-4799-85ab-456c19c86f11\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") "
Sep 09 21:42:31.294398 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294344    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-model-cache\") pod \"393b0a04-fb77-4799-85ab-456c19c86f11\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") "
Sep 09 21:42:31.294398 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294382    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/393b0a04-fb77-4799-85ab-456c19c86f11-cabundle-cert\") pod \"393b0a04-fb77-4799-85ab-456c19c86f11\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") "
Sep 09 21:42:31.294709 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294430    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-kserve-provision-location\") pod \"393b0a04-fb77-4799-85ab-456c19c86f11\" (UID: \"393b0a04-fb77-4799-85ab-456c19c86f11\") "
Sep 09 21:42:31.295055 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294914    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-model-cache" (OuterVolumeSpecName: "model-cache") pod "393b0a04-fb77-4799-85ab-456c19c86f11" (UID: "393b0a04-fb77-4799-85ab-456c19c86f11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:42:31.295055 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.294999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-home" (OuterVolumeSpecName: "home") pod "393b0a04-fb77-4799-85ab-456c19c86f11" (UID: "393b0a04-fb77-4799-85ab-456c19c86f11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:42:31.295055 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.295022    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393b0a04-fb77-4799-85ab-456c19c86f11-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "393b0a04-fb77-4799-85ab-456c19c86f11" (UID: "393b0a04-fb77-4799-85ab-456c19c86f11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:42:31.297320 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.297296    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-dshm" (OuterVolumeSpecName: "dshm") pod "393b0a04-fb77-4799-85ab-456c19c86f11" (UID: "393b0a04-fb77-4799-85ab-456c19c86f11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:42:31.297424 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.297324    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393b0a04-fb77-4799-85ab-456c19c86f11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "393b0a04-fb77-4799-85ab-456c19c86f11" (UID: "393b0a04-fb77-4799-85ab-456c19c86f11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:42:31.297472 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.297429    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393b0a04-fb77-4799-85ab-456c19c86f11-kube-api-access-mwqtp" (OuterVolumeSpecName: "kube-api-access-mwqtp") pod "393b0a04-fb77-4799-85ab-456c19c86f11" (UID: "393b0a04-fb77-4799-85ab-456c19c86f11"). InnerVolumeSpecName "kube-api-access-mwqtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:42:31.312526 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.312496    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "393b0a04-fb77-4799-85ab-456c19c86f11" (UID: "393b0a04-fb77-4799-85ab-456c19c86f11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:42:31.319151 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.319128    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "393b0a04-fb77-4799-85ab-456c19c86f11" (UID: "393b0a04-fb77-4799-85ab-456c19c86f11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:42:31.395617 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.395582    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwqtp\" (UniqueName: \"kubernetes.io/projected/393b0a04-fb77-4799-85ab-456c19c86f11-kube-api-access-mwqtp\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:42:31.395617 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.395615    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393b0a04-fb77-4799-85ab-456c19c86f11-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:42:31.395617 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.395624    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:42:31.395617 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.395632    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:42:31.395617 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.395639    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:42:31.395920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.395647    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/393b0a04-fb77-4799-85ab-456c19c86f11-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:42:31.395920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.395656    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:42:31.395920 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.395665    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393b0a04-fb77-4799-85ab-456c19c86f11-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:42:31.915907 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.915866    2583 generic.go:356] "Generic (PLEG): container finished" podID="393b0a04-fb77-4799-85ab-456c19c86f11" containerID="734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87" exitCode=0
Sep 09 21:42:31.916291 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.915930    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"
Sep 09 21:42:31.916291 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.915954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" event={"ID":"393b0a04-fb77-4799-85ab-456c19c86f11","Type":"ContainerDied","Data":"734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87"}
Sep 09 21:42:31.916291 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.916012    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz" event={"ID":"393b0a04-fb77-4799-85ab-456c19c86f11","Type":"ContainerDied","Data":"57d48997f3de316ae3bddfc19ad7fb8e2ff9b3c824b7ad31137188e1b5c72ec8"}
Sep 09 21:42:31.916291 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.916036    2583 scope.go:117] "RemoveContainer" containerID="734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87"
Sep 09 21:42:31.924843 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.924828    2583 scope.go:117] "RemoveContainer" containerID="64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186"
Sep 09 21:42:31.932751 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.932732    2583 scope.go:117] "RemoveContainer" containerID="734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87"
Sep 09 21:42:31.933044 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:42:31.933023    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87\": container with ID starting with 734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87 not found: ID does not exist" containerID="734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87"
Sep 09 21:42:31.933150 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.933057    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87"} err="failed to get container status \"734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87\": rpc error: code = NotFound desc = could not find container \"734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87\": container with ID starting with 734260def935e1846db479b3ed3922eb94204e7c411287b18d3a63344cec4c87 not found: ID does not exist"
Sep 09 21:42:31.933150 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.933081    2583 scope.go:117] "RemoveContainer" containerID="64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186"
Sep 09 21:42:31.933753 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:42:31.933476    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186\": container with ID starting with 64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186 not found: ID does not exist" containerID="64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186"
Sep 09 21:42:31.933753 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.933516    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186"} err="failed to get container status \"64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186\": rpc error: code = NotFound desc = could not find container \"64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186\": container with ID starting with 64371887ecf86b8a699d77feb3546fbc4297decd59631b6da7bb10a29b66d186 not found: ID does not exist"
Sep 09 21:42:31.934853 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.934830    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"]
Sep 09 21:42:31.938073 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:31.938050    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6f4f6f877d827hz"]
Sep 09 21:42:33.866000 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:42:33.865957    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" path="/var/lib/kubelet/pods/393b0a04-fb77-4799-85ab-456c19c86f11/volumes"
Sep 09 21:43:20.787003 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.786962    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"]
Sep 09 21:43:20.787555 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.787354    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="storage-initializer"
Sep 09 21:43:20.787555 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.787366    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="storage-initializer"
Sep 09 21:43:20.787555 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.787381    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main"
Sep 09 21:43:20.787555 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.787386    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main"
Sep 09 21:43:20.787555 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.787439    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="393b0a04-fb77-4799-85ab-456c19c86f11" containerName="main"
Sep 09 21:43:20.790420 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.790403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.793029 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.792996    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 09 21:43:20.793190 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.793131    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 09 21:43:20.793190 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.793159    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 09 21:43:20.793326 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.793285    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-86tn8\""
Sep 09 21:43:20.800266 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.800244    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"]
Sep 09 21:43:20.834815 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.834774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.834815 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.834812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.835060 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.834831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8jht\" (UniqueName: \"kubernetes.io/projected/f4da7ea9-9929-4cb7-811d-06055e887d0c-kube-api-access-p8jht\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.835060 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.834945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.835060 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.834995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4da7ea9-9929-4cb7-811d-06055e887d0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.835219 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.835060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.835219 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.835084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936294 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936294 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936574 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8jht\" (UniqueName: \"kubernetes.io/projected/f4da7ea9-9929-4cb7-811d-06055e887d0c-kube-api-access-p8jht\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936743 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936743 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4da7ea9-9929-4cb7-811d-06055e887d0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936743 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936743 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936716    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936919 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.936972 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.936925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.938919 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.938896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.939166 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.939150    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4da7ea9-9929-4cb7-811d-06055e887d0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:20.945602 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:20.945576    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8jht\" (UniqueName: \"kubernetes.io/projected/f4da7ea9-9929-4cb7-811d-06055e887d0c-kube-api-access-p8jht\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:21.101755 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:21.101705    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:21.380627 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:21.380541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"]
Sep 09 21:43:21.429096 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:21.429074    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:43:22.154008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:22.153963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" event={"ID":"f4da7ea9-9929-4cb7-811d-06055e887d0c","Type":"ContainerStarted","Data":"24197b74e74d1fd28517cd090f2c4c724c3dffcdc5197b0ed9a13acd3e9e1160"}
Sep 09 21:43:29.191876 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:29.191835    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" event={"ID":"f4da7ea9-9929-4cb7-811d-06055e887d0c","Type":"ContainerStarted","Data":"4932630903ffe184316ca6f722425bab2a8ff2f98fea14de7e913f4e35fbf538"}
Sep 09 21:43:30.198897 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:30.198862    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerID="4932630903ffe184316ca6f722425bab2a8ff2f98fea14de7e913f4e35fbf538" exitCode=0
Sep 09 21:43:30.199398 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:30.198935    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" event={"ID":"f4da7ea9-9929-4cb7-811d-06055e887d0c","Type":"ContainerDied","Data":"4932630903ffe184316ca6f722425bab2a8ff2f98fea14de7e913f4e35fbf538"}
Sep 09 21:43:31.206802 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:31.206704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" event={"ID":"f4da7ea9-9929-4cb7-811d-06055e887d0c","Type":"ContainerStarted","Data":"f0103e486c97303ef245daf7f483247ce5a069b4280fe394ac61f8979b3d0885"}
Sep 09 21:43:31.207204 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:31.206820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" event={"ID":"f4da7ea9-9929-4cb7-811d-06055e887d0c","Type":"ContainerStarted","Data":"d718cfdafbd40e835ca2ecf9fbaa5f3b0f6b7598e36f4acc66934eda4f401663"}
Sep 09 21:43:31.229392 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:31.229323    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podStartSLOduration=2.5617012470000002 podStartE2EDuration="11.229302021s" totalImagesPullingTime="7.283894705s" totalInitContainerRuntime="1.383706069s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:31.225984871 +0000 UTC m=+3049.850458803" watchObservedRunningTime="2026-09-09 21:43:31.229302021 +0000 UTC m=+3049.853775964"
Sep 09 21:43:32.102066 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:32.102026    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:41.102677 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:41.102591    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:43:41.104174 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:41.104146    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:43:51.102691 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:43:51.102645    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:44:01.102700 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:44:01.102655    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:44:11.102617 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:44:11.102566    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:44:21.102481 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:44:21.102440    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:44:31.102311 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:44:31.102266    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:44:41.102809 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:44:41.102761    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:44:51.102580 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:44:51.102534    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:45:01.102835 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:01.102789    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:45:11.102566 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:11.102470    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:45:21.103250 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:21.103202    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:45:31.112603 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:31.112564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:45:31.120377 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:31.120348    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:45:35.716596 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:35.716548    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"]
Sep 09 21:45:35.717708 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:35.717369    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="modelcar" containerID="cri-o://f0103e486c97303ef245daf7f483247ce5a069b4280fe394ac61f8979b3d0885" gracePeriod=60
Sep 09 21:45:35.853661 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:35.853632    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6_f4da7ea9-9929-4cb7-811d-06055e887d0c/modelcar/0.log"
Sep 09 21:45:35.854381 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:35.854358    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerID="f0103e486c97303ef245daf7f483247ce5a069b4280fe394ac61f8979b3d0885" exitCode=143
Sep 09 21:45:35.854476 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:35.854412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" event={"ID":"f4da7ea9-9929-4cb7-811d-06055e887d0c","Type":"ContainerDied","Data":"f0103e486c97303ef245daf7f483247ce5a069b4280fe394ac61f8979b3d0885"}
Sep 09 21:45:50.743062 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:50.743018    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" containerID="cri-o://d718cfdafbd40e835ca2ecf9fbaa5f3b0f6b7598e36f4acc66934eda4f401663" gracePeriod=45
Sep 09 21:45:51.113168 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:51.113126    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 21:45:51.930624 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:51.930592    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6_f4da7ea9-9929-4cb7-811d-06055e887d0c/modelcar/0.log"
Sep 09 21:45:51.931367 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:51.931337    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerID="d718cfdafbd40e835ca2ecf9fbaa5f3b0f6b7598e36f4acc66934eda4f401663" exitCode=0
Sep 09 21:45:51.931496 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:51.931400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" event={"ID":"f4da7ea9-9929-4cb7-811d-06055e887d0c","Type":"ContainerDied","Data":"d718cfdafbd40e835ca2ecf9fbaa5f3b0f6b7598e36f4acc66934eda4f401663"}
Sep 09 21:45:51.931496 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:51.931458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6" event={"ID":"f4da7ea9-9929-4cb7-811d-06055e887d0c","Type":"ContainerDied","Data":"24197b74e74d1fd28517cd090f2c4c724c3dffcdc5197b0ed9a13acd3e9e1160"}
Sep 09 21:45:51.931496 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:51.931487    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24197b74e74d1fd28517cd090f2c4c724c3dffcdc5197b0ed9a13acd3e9e1160"
Sep 09 21:45:52.009417 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.009359    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6_f4da7ea9-9929-4cb7-811d-06055e887d0c/modelcar/0.log"
Sep 09 21:45:52.012526 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.012501    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:45:52.128868 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.128832    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-model-cache\") pod \"f4da7ea9-9929-4cb7-811d-06055e887d0c\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") "
Sep 09 21:45:52.128868 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.128871    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4da7ea9-9929-4cb7-811d-06055e887d0c-tls-certs\") pod \"f4da7ea9-9929-4cb7-811d-06055e887d0c\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") "
Sep 09 21:45:52.129261 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.128902    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-kserve-provision-location\") pod \"f4da7ea9-9929-4cb7-811d-06055e887d0c\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") "
Sep 09 21:45:52.129261 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.128928    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-home\") pod \"f4da7ea9-9929-4cb7-811d-06055e887d0c\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") "
Sep 09 21:45:52.129261 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.128961    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8jht\" (UniqueName: \"kubernetes.io/projected/f4da7ea9-9929-4cb7-811d-06055e887d0c-kube-api-access-p8jht\") pod \"f4da7ea9-9929-4cb7-811d-06055e887d0c\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") "
Sep 09 21:45:52.129261 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.129010    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-dshm\") pod \"f4da7ea9-9929-4cb7-811d-06055e887d0c\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") "
Sep 09 21:45:52.129261 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.129072    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-tmp-dir\") pod \"f4da7ea9-9929-4cb7-811d-06055e887d0c\" (UID: \"f4da7ea9-9929-4cb7-811d-06055e887d0c\") "
Sep 09 21:45:52.129261 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.129173    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4da7ea9-9929-4cb7-811d-06055e887d0c" (UID: "f4da7ea9-9929-4cb7-811d-06055e887d0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:45:52.129544 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.129303    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4da7ea9-9929-4cb7-811d-06055e887d0c" (UID: "f4da7ea9-9929-4cb7-811d-06055e887d0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:45:52.129544 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.129447    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:45:52.129544 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.129467    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:45:52.129913 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.129885    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-home" (OuterVolumeSpecName: "home") pod "f4da7ea9-9929-4cb7-811d-06055e887d0c" (UID: "f4da7ea9-9929-4cb7-811d-06055e887d0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:45:52.131491 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.131455    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-dshm" (OuterVolumeSpecName: "dshm") pod "f4da7ea9-9929-4cb7-811d-06055e887d0c" (UID: "f4da7ea9-9929-4cb7-811d-06055e887d0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:45:52.131607 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.131558    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4da7ea9-9929-4cb7-811d-06055e887d0c-kube-api-access-p8jht" (OuterVolumeSpecName: "kube-api-access-p8jht") pod "f4da7ea9-9929-4cb7-811d-06055e887d0c" (UID: "f4da7ea9-9929-4cb7-811d-06055e887d0c"). InnerVolumeSpecName "kube-api-access-p8jht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:45:52.131607 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.131581    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4da7ea9-9929-4cb7-811d-06055e887d0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4da7ea9-9929-4cb7-811d-06055e887d0c" (UID: "f4da7ea9-9929-4cb7-811d-06055e887d0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:45:52.141147 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.141118    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4da7ea9-9929-4cb7-811d-06055e887d0c" (UID: "f4da7ea9-9929-4cb7-811d-06055e887d0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:45:52.230887 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.230848    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:45:52.230887 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.230882    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8jht\" (UniqueName: \"kubernetes.io/projected/f4da7ea9-9929-4cb7-811d-06055e887d0c-kube-api-access-p8jht\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:45:52.230887 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.230892    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:45:52.231232 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.230924    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4da7ea9-9929-4cb7-811d-06055e887d0c-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:45:52.231232 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.230933    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4da7ea9-9929-4cb7-811d-06055e887d0c-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:45:52.936577 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.936543    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"
Sep 09 21:45:52.957968 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.957926    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"]
Sep 09 21:45:52.961417 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:52.961388    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fklcj6"]
Sep 09 21:45:53.865294 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:45:53.865259    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" path="/var/lib/kubelet/pods/f4da7ea9-9929-4cb7-811d-06055e887d0c/volumes"
Sep 09 21:48:05.820376 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.820294    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"]
Sep 09 21:48:05.820812 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.820665    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main"
Sep 09 21:48:05.820812 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.820676    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main"
Sep 09 21:48:05.820812 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.820687    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="modelcar"
Sep 09 21:48:05.820812 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.820692    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="modelcar"
Sep 09 21:48:05.820812 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.820709    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="modelcar-init"
Sep 09 21:48:05.820812 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.820714    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="modelcar-init"
Sep 09 21:48:05.820812 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.820772    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="main"
Sep 09 21:48:05.820812 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.820779    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4da7ea9-9929-4cb7-811d-06055e887d0c" containerName="modelcar"
Sep 09 21:48:05.823814 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.823790    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:05.827542 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.827516    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 09 21:48:05.827667 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.827592    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 09 21:48:05.827667 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.827639    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-drz9b\""
Sep 09 21:48:05.827796 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.827683    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 09 21:48:05.827846 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.827793    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-97ft5\""
Sep 09 21:48:05.832989 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.832966    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"]
Sep 09 21:48:05.919482 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.919430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgzw\" (UniqueName: \"kubernetes.io/projected/5686f1c7-9145-4d46-a349-1d1ce9c059ad-kube-api-access-xtgzw\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj\" (UID: \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:05.919738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:05.919632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj\" (UID: \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:06.020434 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:06.020397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj\" (UID: \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:06.020434 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:06.020438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgzw\" (UniqueName: \"kubernetes.io/projected/5686f1c7-9145-4d46-a349-1d1ce9c059ad-kube-api-access-xtgzw\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj\" (UID: \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:06.023174 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:06.023148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj\" (UID: \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:06.031212 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:06.031184    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgzw\" (UniqueName: \"kubernetes.io/projected/5686f1c7-9145-4d46-a349-1d1ce9c059ad-kube-api-access-xtgzw\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj\" (UID: \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:06.135462 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:06.135363    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:06.407594 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:06.407567    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"]
Sep 09 21:48:06.410300 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:48:06.410273    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5686f1c7_9145_4d46_a349_1d1ce9c059ad.slice/crio-0fe958a9dc4fd48b754f9814b8bc24604f4424291fbd313b3667dbc8f4c34dc2 WatchSource:0}: Error finding container 0fe958a9dc4fd48b754f9814b8bc24604f4424291fbd313b3667dbc8f4c34dc2: Status 404 returned error can't find the container with id 0fe958a9dc4fd48b754f9814b8bc24604f4424291fbd313b3667dbc8f4c34dc2
Sep 09 21:48:06.579919 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:06.579879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" event={"ID":"5686f1c7-9145-4d46-a349-1d1ce9c059ad","Type":"ContainerStarted","Data":"0fe958a9dc4fd48b754f9814b8bc24604f4424291fbd313b3667dbc8f4c34dc2"}
Sep 09 21:48:08.591693 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:08.591657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" event={"ID":"5686f1c7-9145-4d46-a349-1d1ce9c059ad","Type":"ContainerStarted","Data":"0cd0a923183b2569805c1ef08998213d279f7eb734ef3952a4be9000ba3f69d2"}
Sep 09 21:48:08.592099 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:08.591769    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:08.608712 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:08.608660    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" podStartSLOduration=2.009716961 podStartE2EDuration="3.608642932s" totalImagesPullingTime="1.598925971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:08.607427539 +0000 UTC m=+3327.231901493" watchObservedRunningTime="2026-09-09 21:48:08.608642932 +0000 UTC m=+3327.233116872"
Sep 09 21:48:09.314988 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:09.314958    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"]
Sep 09 21:48:09.596951 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:09.596859    2583 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" secret="" err="secret \"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-drz9b\" not found"
Sep 09 21:48:09.656830 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:09.656788    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:09.657021 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:09.656870    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs podName:5686f1c7-9145-4d46-a349-1d1ce9c059ad nodeName:}" failed. No retries permitted until 2026-09-09 21:48:10.156850836 +0000 UTC m=+3328.781324773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" (UID: "5686f1c7-9145-4d46-a349-1d1ce9c059ad") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:10.162643 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:10.162602    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:10.162838 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:10.162695    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs podName:5686f1c7-9145-4d46-a349-1d1ce9c059ad nodeName:}" failed. No retries permitted until 2026-09-09 21:48:11.162676119 +0000 UTC m=+3329.787150040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" (UID: "5686f1c7-9145-4d46-a349-1d1ce9c059ad") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:11.172290 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:11.172257    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:11.172681 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:11.172337    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs podName:5686f1c7-9145-4d46-a349-1d1ce9c059ad nodeName:}" failed. No retries permitted until 2026-09-09 21:48:13.172318769 +0000 UTC m=+3331.796792687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" (UID: "5686f1c7-9145-4d46-a349-1d1ce9c059ad") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:13.191477 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:13.191442    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:13.191859 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:13.191525    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs podName:5686f1c7-9145-4d46-a349-1d1ce9c059ad nodeName:}" failed. No retries permitted until 2026-09-09 21:48:17.191508098 +0000 UTC m=+3335.815982021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" (UID: "5686f1c7-9145-4d46-a349-1d1ce9c059ad") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:17.229212 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:17.229174    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:17.229629 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:17.229272    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs podName:5686f1c7-9145-4d46-a349-1d1ce9c059ad nodeName:}" failed. No retries permitted until 2026-09-09 21:48:25.229256173 +0000 UTC m=+3343.853730105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" (UID: "5686f1c7-9145-4d46-a349-1d1ce9c059ad") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:25.305713 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:25.305669    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:25.306156 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:25.305759    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs podName:5686f1c7-9145-4d46-a349-1d1ce9c059ad nodeName:}" failed. No retries permitted until 2026-09-09 21:48:41.305741977 +0000 UTC m=+3359.930215896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" (UID: "5686f1c7-9145-4d46-a349-1d1ce9c059ad") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:25.623054 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:25.622959    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" podUID="5686f1c7-9145-4d46-a349-1d1ce9c059ad" containerName="main" containerID="cri-o://0cd0a923183b2569805c1ef08998213d279f7eb734ef3952a4be9000ba3f69d2" gracePeriod=45
Sep 09 21:48:40.606004 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:40.605966    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" podUID="5686f1c7-9145-4d46-a349-1d1ce9c059ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:48:41.357090 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:41.357049    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:41.357280 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:48:41.357157    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs podName:5686f1c7-9145-4d46-a349-1d1ce9c059ad nodeName:}" failed. No retries permitted until 2026-09-09 21:49:13.357141876 +0000 UTC m=+3391.981615795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" (UID: "5686f1c7-9145-4d46-a349-1d1ce9c059ad") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 09 21:48:50.603870 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:50.603828    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" podUID="5686f1c7-9145-4d46-a349-1d1ce9c059ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:48:55.816690 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:55.816652    2583 generic.go:356] "Generic (PLEG): container finished" podID="5686f1c7-9145-4d46-a349-1d1ce9c059ad" containerID="0cd0a923183b2569805c1ef08998213d279f7eb734ef3952a4be9000ba3f69d2" exitCode=0
Sep 09 21:48:55.817225 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:55.816735    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" event={"ID":"5686f1c7-9145-4d46-a349-1d1ce9c059ad","Type":"ContainerDied","Data":"0cd0a923183b2569805c1ef08998213d279f7eb734ef3952a4be9000ba3f69d2"}
Sep 09 21:48:55.869157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:55.869127    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:56.001084 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.000981    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtgzw\" (UniqueName: \"kubernetes.io/projected/5686f1c7-9145-4d46-a349-1d1ce9c059ad-kube-api-access-xtgzw\") pod \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\" (UID: \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\") "
Sep 09 21:48:56.001273 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.001141    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs\") pod \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\" (UID: \"5686f1c7-9145-4d46-a349-1d1ce9c059ad\") "
Sep 09 21:48:56.003497 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.003466    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5686f1c7-9145-4d46-a349-1d1ce9c059ad" (UID: "5686f1c7-9145-4d46-a349-1d1ce9c059ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:56.003618 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.003500    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5686f1c7-9145-4d46-a349-1d1ce9c059ad-kube-api-access-xtgzw" (OuterVolumeSpecName: "kube-api-access-xtgzw") pod "5686f1c7-9145-4d46-a349-1d1ce9c059ad" (UID: "5686f1c7-9145-4d46-a349-1d1ce9c059ad"). InnerVolumeSpecName "kube-api-access-xtgzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:56.102309 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.102269    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5686f1c7-9145-4d46-a349-1d1ce9c059ad-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:48:56.102309 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.102301    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtgzw\" (UniqueName: \"kubernetes.io/projected/5686f1c7-9145-4d46-a349-1d1ce9c059ad-kube-api-access-xtgzw\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:48:56.825130 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.825074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj" event={"ID":"5686f1c7-9145-4d46-a349-1d1ce9c059ad","Type":"ContainerDied","Data":"0fe958a9dc4fd48b754f9814b8bc24604f4424291fbd313b3667dbc8f4c34dc2"}
Sep 09 21:48:56.825130 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.825126    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"
Sep 09 21:48:56.825577 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.825174    2583 scope.go:117] "RemoveContainer" containerID="0cd0a923183b2569805c1ef08998213d279f7eb734ef3952a4be9000ba3f69d2"
Sep 09 21:48:56.847204 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.847124    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"]
Sep 09 21:48:56.849336 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:56.849308    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddgctsj"]
Sep 09 21:48:57.865404 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:48:57.865372    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5686f1c7-9145-4d46-a349-1d1ce9c059ad" path="/var/lib/kubelet/pods/5686f1c7-9145-4d46-a349-1d1ce9c059ad/volumes"
Sep 09 21:49:37.525744 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.525659    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"]
Sep 09 21:49:37.526359 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.526063    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5686f1c7-9145-4d46-a349-1d1ce9c059ad" containerName="main"
Sep 09 21:49:37.526359 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.526077    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5686f1c7-9145-4d46-a349-1d1ce9c059ad" containerName="main"
Sep 09 21:49:37.526359 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.526201    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5686f1c7-9145-4d46-a349-1d1ce9c059ad" containerName="main"
Sep 09 21:49:37.529468 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.529446    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.531964 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.531919    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:49:37.532150 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.531988    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-lhdlj\""
Sep 09 21:49:37.532150 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.532043    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 09 21:49:37.532150 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.532047    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 09 21:49:37.540882 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.540850    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"]
Sep 09 21:49:37.688738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.688705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.688946 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.688755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.688946 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.688779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f72b47-c010-48d2-b442-a91186cda71f-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.688946 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.688799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.688946 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.688854    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.688946 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.688874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.688946 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.688888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f72b47-c010-48d2-b442-a91186cda71f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.688946 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.688918    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.689268 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.688980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fgv\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-kube-api-access-24fgv\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.789825 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.789719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.789825 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.789768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f72b47-c010-48d2-b442-a91186cda71f-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.789825 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.789793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.789857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.789885    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.789917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f72b47-c010-48d2-b442-a91186cda71f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.789952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.789981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fgv\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-kube-api-access-24fgv\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.790088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790454 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.790237    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790454 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.790365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790454 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.790397    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790610 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.790492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.790705 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.790684    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f72b47-c010-48d2-b442-a91186cda71f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.792419 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.792395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.792603 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.792583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f72b47-c010-48d2-b442-a91186cda71f-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.806221 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.806191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.806642 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.806622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fgv\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-kube-api-access-24fgv\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p7tfc\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:37.843893 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:37.843851    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:38.123505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:38.123475    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"]
Sep 09 21:49:38.125662 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:49:38.125626    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f72b47_c010_48d2_b442_a91186cda71f.slice/crio-f5b52972f3a52aba58ff849e74d5e6346639760167bc0f5b89d3485bfa687bc3 WatchSource:0}: Error finding container f5b52972f3a52aba58ff849e74d5e6346639760167bc0f5b89d3485bfa687bc3: Status 404 returned error can't find the container with id f5b52972f3a52aba58ff849e74d5e6346639760167bc0f5b89d3485bfa687bc3
Sep 09 21:49:38.127638 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:38.127607    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:49:38.127705 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:38.127678    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:49:38.127748 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:38.127708    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:49:39.032348 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:39.032311    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" event={"ID":"a7f72b47-c010-48d2-b442-a91186cda71f","Type":"ContainerStarted","Data":"748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380"}
Sep 09 21:49:39.032736 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:39.032364    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" event={"ID":"a7f72b47-c010-48d2-b442-a91186cda71f","Type":"ContainerStarted","Data":"f5b52972f3a52aba58ff849e74d5e6346639760167bc0f5b89d3485bfa687bc3"}
Sep 09 21:49:39.053531 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:39.053482    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" podStartSLOduration=2.053464235 podStartE2EDuration="2.053464235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:39.051260058 +0000 UTC m=+3417.675733991" watchObservedRunningTime="2026-09-09 21:49:39.053464235 +0000 UTC m=+3417.677938176"
Sep 09 21:49:39.844735 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:39.844697    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:39.849864 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:39.849835    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:40.037361 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:40.037327    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:40.038461 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:40.038442    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:42.282557 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:42.282521    2583 scope.go:117] "RemoveContainer" containerID="4932630903ffe184316ca6f722425bab2a8ff2f98fea14de7e913f4e35fbf538"
Sep 09 21:49:42.291086 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:42.291062    2583 scope.go:117] "RemoveContainer" containerID="f0103e486c97303ef245daf7f483247ce5a069b4280fe394ac61f8979b3d0885"
Sep 09 21:49:42.298875 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:42.298855    2583 scope.go:117] "RemoveContainer" containerID="d718cfdafbd40e835ca2ecf9fbaa5f3b0f6b7598e36f4acc66934eda4f401663"
Sep 09 21:49:51.095326 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:51.095292    2583 generic.go:356] "Generic (PLEG): container finished" podID="a7f72b47-c010-48d2-b442-a91186cda71f" containerID="748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380" exitCode=0
Sep 09 21:49:51.096149 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:51.095342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" event={"ID":"a7f72b47-c010-48d2-b442-a91186cda71f","Type":"ContainerDied","Data":"748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380"}
Sep 09 21:49:51.096149 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:51.095845    2583 scope.go:117] "RemoveContainer" containerID="748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380"
Sep 09 21:49:51.096972 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:51.096942    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:49:51.097023 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:51.097013    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:49:51.097070 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:51.097055    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 21:49:52.101827 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:52.101791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" event={"ID":"a7f72b47-c010-48d2-b442-a91186cda71f","Type":"ContainerStarted","Data":"546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963"}
Sep 09 21:49:52.844722 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:52.844679    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:52.845965 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:52.845928    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 09 21:49:53.845141 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:53.845085    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 09 21:49:54.844626 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:54.844575    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 09 21:49:55.037666 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:55.037617    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:55.844525 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:55.844483    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 09 21:49:56.845308 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:56.845268    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 09 21:49:57.848480 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:57.848449    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:49:57.849330 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:49:57.849309    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:50:29.584921 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.584888    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"]
Sep 09 21:50:29.588653 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.588628    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.591394 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.591301    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 09 21:50:29.591549 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.591421    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 09 21:50:29.591549 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.591493    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-lnjzv\""
Sep 09 21:50:29.591682 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.591551    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:50:29.591682 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.591563    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 09 21:50:29.591825 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.591800    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 09 21:50:29.600506 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.600478    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"]
Sep 09 21:50:29.688264 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.688227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c8d985-29a8-4946-af20-3c6dabb3318a-tls-certs\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.688458 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.688275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-home\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.688458 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.688301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8d985-29a8-4946-af20-3c6dabb3318a-cabundle-cert\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.688458 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.688331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-dshm\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.688458 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.688352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-tmp-dir\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.688458 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.688369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-kserve-provision-location\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.688458 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.688441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9jv4\" (UniqueName: \"kubernetes.io/projected/e4c8d985-29a8-4946-af20-3c6dabb3318a-kube-api-access-q9jv4\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.688673 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.688506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-model-cache\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.759388 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.759349    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"]
Sep 09 21:50:29.769434 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.769361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:50:29.771890 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.771862    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-j2dq2\""
Sep 09 21:50:29.772406 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.772381    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"]
Sep 09 21:50:29.789532 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.789502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c8d985-29a8-4946-af20-3c6dabb3318a-tls-certs\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.789674 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.789546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-home\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.789674 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.789563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8d985-29a8-4946-af20-3c6dabb3318a-cabundle-cert\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.789674 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.789580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-dshm\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.789674 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.789595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-tmp-dir\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.789674 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.789611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-kserve-provision-location\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.789952 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.789739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362f05ba-bdd3-42c8-b52f-b244d102ca99-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5\" (UID: \"362f05ba-bdd3-42c8-b52f-b244d102ca99\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:50:29.790021 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.789983    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-home\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.790076 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.790014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-tmp-dir\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.790076 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.790058    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-kserve-provision-location\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.790246 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.790210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9jv4\" (UniqueName: \"kubernetes.io/projected/e4c8d985-29a8-4946-af20-3c6dabb3318a-kube-api-access-q9jv4\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.790448 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.790422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8d985-29a8-4946-af20-3c6dabb3318a-cabundle-cert\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.790544 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.790509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-model-cache\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.790619 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.790601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr67d\" (UniqueName: \"kubernetes.io/projected/362f05ba-bdd3-42c8-b52f-b244d102ca99-kube-api-access-dr67d\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5\" (UID: \"362f05ba-bdd3-42c8-b52f-b244d102ca99\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:50:29.790833 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.790812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-model-cache\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.792281 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.792254    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-dshm\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.792522 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.792505    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c8d985-29a8-4946-af20-3c6dabb3318a-tls-certs\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.798373 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.798351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9jv4\" (UniqueName: \"kubernetes.io/projected/e4c8d985-29a8-4946-af20-3c6dabb3318a-kube-api-access-q9jv4\") pod \"stop-feature-test-kserve-7644fc97d-w74fl\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:29.891284 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.891187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr67d\" (UniqueName: \"kubernetes.io/projected/362f05ba-bdd3-42c8-b52f-b244d102ca99-kube-api-access-dr67d\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5\" (UID: \"362f05ba-bdd3-42c8-b52f-b244d102ca99\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:50:29.891284 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.891278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362f05ba-bdd3-42c8-b52f-b244d102ca99-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5\" (UID: \"362f05ba-bdd3-42c8-b52f-b244d102ca99\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:50:29.894209 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.894174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362f05ba-bdd3-42c8-b52f-b244d102ca99-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5\" (UID: \"362f05ba-bdd3-42c8-b52f-b244d102ca99\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:50:29.899962 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.899941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr67d\" (UniqueName: \"kubernetes.io/projected/362f05ba-bdd3-42c8-b52f-b244d102ca99-kube-api-access-dr67d\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5\" (UID: \"362f05ba-bdd3-42c8-b52f-b244d102ca99\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:50:29.902870 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:29.902849    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:30.080982 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:30.080942    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:50:30.185132 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:30.185084    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"]
Sep 09 21:50:30.186862 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:50:30.186837    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c8d985_29a8_4946_af20_3c6dabb3318a.slice/crio-66847f5566fc5f7108dbe4e83b9a72589f8554bbbc7d4958cebc8256425ff0e7 WatchSource:0}: Error finding container 66847f5566fc5f7108dbe4e83b9a72589f8554bbbc7d4958cebc8256425ff0e7: Status 404 returned error can't find the container with id 66847f5566fc5f7108dbe4e83b9a72589f8554bbbc7d4958cebc8256425ff0e7
Sep 09 21:50:30.293925 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:30.293885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" event={"ID":"e4c8d985-29a8-4946-af20-3c6dabb3318a","Type":"ContainerStarted","Data":"284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15"}
Sep 09 21:50:30.294093 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:30.293948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" event={"ID":"e4c8d985-29a8-4946-af20-3c6dabb3318a","Type":"ContainerStarted","Data":"66847f5566fc5f7108dbe4e83b9a72589f8554bbbc7d4958cebc8256425ff0e7"}
Sep 09 21:50:30.359546 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:30.359513    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"]
Sep 09 21:50:30.362859 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:50:30.362827    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362f05ba_bdd3_42c8_b52f_b244d102ca99.slice/crio-b69c04e9b3dc5fa335e02be4b3d25920d334407c9a7e61fcc29fb401c7415cca WatchSource:0}: Error finding container b69c04e9b3dc5fa335e02be4b3d25920d334407c9a7e61fcc29fb401c7415cca: Status 404 returned error can't find the container with id b69c04e9b3dc5fa335e02be4b3d25920d334407c9a7e61fcc29fb401c7415cca
Sep 09 21:50:31.300519 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:31.300470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" event={"ID":"362f05ba-bdd3-42c8-b52f-b244d102ca99","Type":"ContainerStarted","Data":"64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69"}
Sep 09 21:50:31.300925 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:31.300532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" event={"ID":"362f05ba-bdd3-42c8-b52f-b244d102ca99","Type":"ContainerStarted","Data":"b69c04e9b3dc5fa335e02be4b3d25920d334407c9a7e61fcc29fb401c7415cca"}
Sep 09 21:50:31.300925 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:31.300619    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:50:31.318725 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:31.318659    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" podStartSLOduration=2.318634155 podStartE2EDuration="2.318634155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:31.316785145 +0000 UTC m=+3469.941259114" watchObservedRunningTime="2026-09-09 21:50:31.318634155 +0000 UTC m=+3469.943108097"
Sep 09 21:50:33.312414 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:33.312378    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerID="284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15" exitCode=0
Sep 09 21:50:33.312948 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:33.312450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" event={"ID":"e4c8d985-29a8-4946-af20-3c6dabb3318a","Type":"ContainerDied","Data":"284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15"}
Sep 09 21:50:34.319633 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:34.319595    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" event={"ID":"e4c8d985-29a8-4946-af20-3c6dabb3318a","Type":"ContainerStarted","Data":"191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14"}
Sep 09 21:50:34.344218 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:34.344158    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podStartSLOduration=3.120308199 podStartE2EDuration="5.344135718s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.223827519s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:34.341986842 +0000 UTC m=+3472.966460784" watchObservedRunningTime="2026-09-09 21:50:34.344135718 +0000 UTC m=+3472.968609660"
Sep 09 21:50:34.903443 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:34.903408    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:39.903974 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:39.903914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:50:39.905394 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:39.905363    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:50:49.904510 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:49.904462    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:50:59.904241 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:50:59.904188    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:51:02.308008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:51:02.307967    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:51:09.904024 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:51:09.903974    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:51:19.904239 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:51:19.904187    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:51:29.903814 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:51:29.903762    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:51:39.903687 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:51:39.903645    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:51:49.903632 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:51:49.903595    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:51:59.903831 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:51:59.903785    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:52:09.903623 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:09.903573    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:52:19.903608 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:19.903568    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 09 21:52:29.915053 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:29.915011    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:52:29.923461 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:29.923432    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:52:31.428376 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:31.427151    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"]
Sep 09 21:52:31.439954 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:31.439917    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"]
Sep 09 21:52:32.311323 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:32.311279    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:52:42.307296 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:42.307254    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:52:46.460707 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:46.460593    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerName="main" containerID="cri-o://64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69" gracePeriod=45
Sep 09 21:52:46.952526 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:46.952480    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main" containerID="cri-o://191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14" gracePeriod=45
Sep 09 21:52:47.708931 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.708904    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:52:47.842137 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842066    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c8d985-29a8-4946-af20-3c6dabb3318a-tls-certs\") pod \"e4c8d985-29a8-4946-af20-3c6dabb3318a\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") "
Sep 09 21:52:47.842355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842165    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-model-cache\") pod \"e4c8d985-29a8-4946-af20-3c6dabb3318a\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") "
Sep 09 21:52:47.842355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842203    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-home\") pod \"e4c8d985-29a8-4946-af20-3c6dabb3318a\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") "
Sep 09 21:52:47.842355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842236    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-tmp-dir\") pod \"e4c8d985-29a8-4946-af20-3c6dabb3318a\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") "
Sep 09 21:52:47.842355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842290    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-dshm\") pod \"e4c8d985-29a8-4946-af20-3c6dabb3318a\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") "
Sep 09 21:52:47.842567 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842434    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-kserve-provision-location\") pod \"e4c8d985-29a8-4946-af20-3c6dabb3318a\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") "
Sep 09 21:52:47.842567 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842447    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4c8d985-29a8-4946-af20-3c6dabb3318a" (UID: "e4c8d985-29a8-4946-af20-3c6dabb3318a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:47.842567 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842480    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9jv4\" (UniqueName: \"kubernetes.io/projected/e4c8d985-29a8-4946-af20-3c6dabb3318a-kube-api-access-q9jv4\") pod \"e4c8d985-29a8-4946-af20-3c6dabb3318a\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") "
Sep 09 21:52:47.842567 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842519    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8d985-29a8-4946-af20-3c6dabb3318a-cabundle-cert\") pod \"e4c8d985-29a8-4946-af20-3c6dabb3318a\" (UID: \"e4c8d985-29a8-4946-af20-3c6dabb3318a\") "
Sep 09 21:52:47.842873 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842855    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:47.842929 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842867    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8d985-29a8-4946-af20-3c6dabb3318a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4c8d985-29a8-4946-af20-3c6dabb3318a" (UID: "e4c8d985-29a8-4946-af20-3c6dabb3318a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:52:47.842985 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.842973    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-home" (OuterVolumeSpecName: "home") pod "e4c8d985-29a8-4946-af20-3c6dabb3318a" (UID: "e4c8d985-29a8-4946-af20-3c6dabb3318a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:47.845368 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.845335    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-dshm" (OuterVolumeSpecName: "dshm") pod "e4c8d985-29a8-4946-af20-3c6dabb3318a" (UID: "e4c8d985-29a8-4946-af20-3c6dabb3318a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:47.845368 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.845345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8d985-29a8-4946-af20-3c6dabb3318a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4c8d985-29a8-4946-af20-3c6dabb3318a" (UID: "e4c8d985-29a8-4946-af20-3c6dabb3318a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:52:47.845544 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.845351    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c8d985-29a8-4946-af20-3c6dabb3318a-kube-api-access-q9jv4" (OuterVolumeSpecName: "kube-api-access-q9jv4") pod "e4c8d985-29a8-4946-af20-3c6dabb3318a" (UID: "e4c8d985-29a8-4946-af20-3c6dabb3318a"). InnerVolumeSpecName "kube-api-access-q9jv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:52:47.860037 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.859992    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4c8d985-29a8-4946-af20-3c6dabb3318a" (UID: "e4c8d985-29a8-4946-af20-3c6dabb3318a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:47.871141 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.871083    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4c8d985-29a8-4946-af20-3c6dabb3318a" (UID: "e4c8d985-29a8-4946-af20-3c6dabb3318a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:47.944155 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.944096    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c8d985-29a8-4946-af20-3c6dabb3318a-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:47.944155 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.944150    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:47.944155 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.944158    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:47.944155 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.944165    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:47.944417 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.944175    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d985-29a8-4946-af20-3c6dabb3318a-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:47.944417 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.944184    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9jv4\" (UniqueName: \"kubernetes.io/projected/e4c8d985-29a8-4946-af20-3c6dabb3318a-kube-api-access-q9jv4\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:47.944417 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:47.944192    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8d985-29a8-4946-af20-3c6dabb3318a-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:48.010075 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.010039    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerID="191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14" exitCode=0
Sep 09 21:52:48.010258 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.010095    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"
Sep 09 21:52:48.010258 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.010129    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" event={"ID":"e4c8d985-29a8-4946-af20-3c6dabb3318a","Type":"ContainerDied","Data":"191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14"}
Sep 09 21:52:48.010258 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.010189    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl" event={"ID":"e4c8d985-29a8-4946-af20-3c6dabb3318a","Type":"ContainerDied","Data":"66847f5566fc5f7108dbe4e83b9a72589f8554bbbc7d4958cebc8256425ff0e7"}
Sep 09 21:52:48.010258 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.010221    2583 scope.go:117] "RemoveContainer" containerID="191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14"
Sep 09 21:52:48.019464 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.019444    2583 scope.go:117] "RemoveContainer" containerID="284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15"
Sep 09 21:52:48.027382 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.027362    2583 scope.go:117] "RemoveContainer" containerID="191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14"
Sep 09 21:52:48.027674 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:52:48.027653    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14\": container with ID starting with 191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14 not found: ID does not exist" containerID="191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14"
Sep 09 21:52:48.027751 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.027686    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14"} err="failed to get container status \"191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14\": rpc error: code = NotFound desc = could not find container \"191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14\": container with ID starting with 191913de1d793c708a7d82d6ddff5680a9dfddab4ffb14f2a0a556e04a53ab14 not found: ID does not exist"
Sep 09 21:52:48.027751 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.027711    2583 scope.go:117] "RemoveContainer" containerID="284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15"
Sep 09 21:52:48.027999 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:52:48.027980    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15\": container with ID starting with 284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15 not found: ID does not exist" containerID="284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15"
Sep 09 21:52:48.028037 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.028004    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15"} err="failed to get container status \"284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15\": rpc error: code = NotFound desc = could not find container \"284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15\": container with ID starting with 284712c2da661cb59c3e31d1c9e72df57bdf84f732bd6d414a270ca5f3e8ee15 not found: ID does not exist"
Sep 09 21:52:48.033515 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.033485    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"]
Sep 09 21:52:48.038297 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:48.038270    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-w74fl"]
Sep 09 21:52:49.865904 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:49.865863    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" path="/var/lib/kubelet/pods/e4c8d985-29a8-4946-af20-3c6dabb3318a/volumes"
Sep 09 21:52:52.307491 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:52.307455    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:52:52.307942 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:52.307554    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:52:53.713016 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:53.712977    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"]
Sep 09 21:52:53.713468 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:53.713196    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy" containerID="cri-o://546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963" gracePeriod=30
Sep 09 21:52:57.851546 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:57.851491    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:52:58.963429 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:58.963401    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:52:59.037707 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037673    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f72b47-c010-48d2-b442-a91186cda71f-istio-podinfo\") pod \"a7f72b47-c010-48d2-b442-a91186cda71f\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") "
Sep 09 21:52:59.037861 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037721    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-socket\") pod \"a7f72b47-c010-48d2-b442-a91186cda71f\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") "
Sep 09 21:52:59.037861 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24fgv\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-kube-api-access-24fgv\") pod \"a7f72b47-c010-48d2-b442-a91186cda71f\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") "
Sep 09 21:52:59.037861 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037784    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f72b47-c010-48d2-b442-a91186cda71f-istiod-ca-cert\") pod \"a7f72b47-c010-48d2-b442-a91186cda71f\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") "
Sep 09 21:52:59.037861 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037831    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-credential-socket\") pod \"a7f72b47-c010-48d2-b442-a91186cda71f\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") "
Sep 09 21:52:59.037861 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037855    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-istio-token\") pod \"a7f72b47-c010-48d2-b442-a91186cda71f\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") "
Sep 09 21:52:59.038215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037886    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-data\") pod \"a7f72b47-c010-48d2-b442-a91186cda71f\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") "
Sep 09 21:52:59.038215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037929    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-certs\") pod \"a7f72b47-c010-48d2-b442-a91186cda71f\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") "
Sep 09 21:52:59.038215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037983    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-envoy\") pod \"a7f72b47-c010-48d2-b442-a91186cda71f\" (UID: \"a7f72b47-c010-48d2-b442-a91186cda71f\") "
Sep 09 21:52:59.038215 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.037999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7f72b47-c010-48d2-b442-a91186cda71f" (UID: "a7f72b47-c010-48d2-b442-a91186cda71f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:59.038432 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.038232    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7f72b47-c010-48d2-b442-a91186cda71f" (UID: "a7f72b47-c010-48d2-b442-a91186cda71f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:59.038432 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.038240    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7f72b47-c010-48d2-b442-a91186cda71f" (UID: "a7f72b47-c010-48d2-b442-a91186cda71f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:59.038432 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.038326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7f72b47-c010-48d2-b442-a91186cda71f" (UID: "a7f72b47-c010-48d2-b442-a91186cda71f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:59.038432 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.038366    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f72b47-c010-48d2-b442-a91186cda71f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7f72b47-c010-48d2-b442-a91186cda71f" (UID: "a7f72b47-c010-48d2-b442-a91186cda71f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:52:59.038738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.038495    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f72b47-c010-48d2-b442-a91186cda71f-istiod-ca-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:59.038738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.038513    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-credential-socket\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:59.038738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.038527    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-data\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:59.038738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.038541    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:59.038738 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.038557    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-workload-socket\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:59.041209 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.041171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7f72b47-c010-48d2-b442-a91186cda71f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7f72b47-c010-48d2-b442-a91186cda71f" (UID: "a7f72b47-c010-48d2-b442-a91186cda71f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:52:59.041329 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.041302    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7f72b47-c010-48d2-b442-a91186cda71f" (UID: "a7f72b47-c010-48d2-b442-a91186cda71f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:59.041449 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.041425    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-kube-api-access-24fgv" (OuterVolumeSpecName: "kube-api-access-24fgv") pod "a7f72b47-c010-48d2-b442-a91186cda71f" (UID: "a7f72b47-c010-48d2-b442-a91186cda71f"). InnerVolumeSpecName "kube-api-access-24fgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:52:59.041540 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.041430    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7f72b47-c010-48d2-b442-a91186cda71f" (UID: "a7f72b47-c010-48d2-b442-a91186cda71f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:52:59.072823 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.072790    2583 generic.go:356] "Generic (PLEG): container finished" podID="a7f72b47-c010-48d2-b442-a91186cda71f" containerID="546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963" exitCode=0
Sep 09 21:52:59.072991 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.072844    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"
Sep 09 21:52:59.072991 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.072876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" event={"ID":"a7f72b47-c010-48d2-b442-a91186cda71f","Type":"ContainerDied","Data":"546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963"}
Sep 09 21:52:59.072991 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.072933    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc" event={"ID":"a7f72b47-c010-48d2-b442-a91186cda71f","Type":"ContainerDied","Data":"f5b52972f3a52aba58ff849e74d5e6346639760167bc0f5b89d3485bfa687bc3"}
Sep 09 21:52:59.072991 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.072957    2583 scope.go:117] "RemoveContainer" containerID="546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963"
Sep 09 21:52:59.082422 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.082401    2583 scope.go:117] "RemoveContainer" containerID="748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380"
Sep 09 21:52:59.092773 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.092732    2583 scope.go:117] "RemoveContainer" containerID="546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963"
Sep 09 21:52:59.093065 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:52:59.093043    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963\": container with ID starting with 546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963 not found: ID does not exist" containerID="546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963"
Sep 09 21:52:59.093211 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.093074    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963"} err="failed to get container status \"546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963\": rpc error: code = NotFound desc = could not find container \"546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963\": container with ID starting with 546b4198fa5bb5bb0e99f75a9dd4714551d95d3f54f27ab645a0585677221963 not found: ID does not exist"
Sep 09 21:52:59.093211 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.093093    2583 scope.go:117] "RemoveContainer" containerID="748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380"
Sep 09 21:52:59.093425 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:52:59.093400    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380\": container with ID starting with 748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380 not found: ID does not exist" containerID="748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380"
Sep 09 21:52:59.093505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.093433    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380"} err="failed to get container status \"748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380\": rpc error: code = NotFound desc = could not find container \"748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380\": container with ID starting with 748820b7d4c3d1d63be68a35d681f1861badd3156b30646661c4eaf414ef0380 not found: ID does not exist"
Sep 09 21:52:59.096405 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.096374    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"]
Sep 09 21:52:59.099956 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.099932    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p7tfc"]
Sep 09 21:52:59.139965 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.139930    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-istio-token\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:59.139965 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.139966    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f72b47-c010-48d2-b442-a91186cda71f-istio-envoy\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:59.140205 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.139979    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f72b47-c010-48d2-b442-a91186cda71f-istio-podinfo\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:59.140205 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.139988    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fgv\" (UniqueName: \"kubernetes.io/projected/a7f72b47-c010-48d2-b442-a91186cda71f-kube-api-access-24fgv\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:52:59.866407 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:52:59.866375    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" path="/var/lib/kubelet/pods/a7f72b47-c010-48d2-b442-a91186cda71f/volumes"
Sep 09 21:53:02.307468 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:02.307430    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:53:12.307689 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:12.307646    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:53:16.715257 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:16.715224    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:53:16.805726 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:16.805692    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr67d\" (UniqueName: \"kubernetes.io/projected/362f05ba-bdd3-42c8-b52f-b244d102ca99-kube-api-access-dr67d\") pod \"362f05ba-bdd3-42c8-b52f-b244d102ca99\" (UID: \"362f05ba-bdd3-42c8-b52f-b244d102ca99\") "
Sep 09 21:53:16.805924 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:16.805760    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362f05ba-bdd3-42c8-b52f-b244d102ca99-tls-certs\") pod \"362f05ba-bdd3-42c8-b52f-b244d102ca99\" (UID: \"362f05ba-bdd3-42c8-b52f-b244d102ca99\") "
Sep 09 21:53:16.808021 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:16.807990    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f05ba-bdd3-42c8-b52f-b244d102ca99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "362f05ba-bdd3-42c8-b52f-b244d102ca99" (UID: "362f05ba-bdd3-42c8-b52f-b244d102ca99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:53:16.808021 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:16.808002    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362f05ba-bdd3-42c8-b52f-b244d102ca99-kube-api-access-dr67d" (OuterVolumeSpecName: "kube-api-access-dr67d") pod "362f05ba-bdd3-42c8-b52f-b244d102ca99" (UID: "362f05ba-bdd3-42c8-b52f-b244d102ca99"). InnerVolumeSpecName "kube-api-access-dr67d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:53:16.906995 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:16.906954    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr67d\" (UniqueName: \"kubernetes.io/projected/362f05ba-bdd3-42c8-b52f-b244d102ca99-kube-api-access-dr67d\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:53:16.906995 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:16.906988    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362f05ba-bdd3-42c8-b52f-b244d102ca99-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:53:17.166156 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.166045    2583 generic.go:356] "Generic (PLEG): container finished" podID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerID="64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69" exitCode=0
Sep 09 21:53:17.166156 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.166126    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"
Sep 09 21:53:17.166156 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.166130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" event={"ID":"362f05ba-bdd3-42c8-b52f-b244d102ca99","Type":"ContainerDied","Data":"64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69"}
Sep 09 21:53:17.166417 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.166182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5" event={"ID":"362f05ba-bdd3-42c8-b52f-b244d102ca99","Type":"ContainerDied","Data":"b69c04e9b3dc5fa335e02be4b3d25920d334407c9a7e61fcc29fb401c7415cca"}
Sep 09 21:53:17.166417 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.166205    2583 scope.go:117] "RemoveContainer" containerID="64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69"
Sep 09 21:53:17.175211 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.175186    2583 scope.go:117] "RemoveContainer" containerID="64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69"
Sep 09 21:53:17.175514 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:53:17.175493    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69\": container with ID starting with 64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69 not found: ID does not exist" containerID="64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69"
Sep 09 21:53:17.175584 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.175524    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69"} err="failed to get container status \"64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69\": rpc error: code = NotFound desc = could not find container \"64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69\": container with ID starting with 64f4c8dd8097d7893a933ee80ee4c114f2338cef9c4d58d4a4f651d7f2740b69 not found: ID does not exist"
Sep 09 21:53:17.189410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.189384    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"]
Sep 09 21:53:17.195086 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.195064    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-dc8v5"]
Sep 09 21:53:17.866421 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:17.866379    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" path="/var/lib/kubelet/pods/362f05ba-bdd3-42c8-b52f-b244d102ca99/volumes"
Sep 09 21:53:33.179569 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179530    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"]
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179912    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="storage-initializer"
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179924    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="storage-initializer"
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179945    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy"
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179951    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy"
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179960    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy"
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179966    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy"
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179972    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main"
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179977    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main"
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179983    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerName="main"
Sep 09 21:53:33.179986 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.179988    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerName="main"
Sep 09 21:53:33.180331 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.180047    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="362f05ba-bdd3-42c8-b52f-b244d102ca99" containerName="main"
Sep 09 21:53:33.180331 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.180055    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c8d985-29a8-4946-af20-3c6dabb3318a" containerName="main"
Sep 09 21:53:33.180331 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.180062    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy"
Sep 09 21:53:33.180331 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.180181    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f72b47-c010-48d2-b442-a91186cda71f" containerName="istio-proxy"
Sep 09 21:53:33.183059 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.183042    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.185845 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.185816    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 09 21:53:33.186032 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.185868    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-8d54f\""
Sep 09 21:53:33.186032 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.185822    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:53:33.186032 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.185923    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 09 21:53:33.186032 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.185827    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 09 21:53:33.186032 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.185822    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 09 21:53:33.194475 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.194450    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"]
Sep 09 21:53:33.258155 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.258092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-dshm\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.258347 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.258175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.258347 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.258205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-cabundle-cert\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.258347 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.258227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tls-certs\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.258347 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.258270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-home\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.258347 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.258327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkl82\" (UniqueName: \"kubernetes.io/projected/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kube-api-access-qkl82\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.258516 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.258352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-model-cache\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.258516 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.258415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tmp-dir\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.360242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tmp-dir\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.360307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-dshm\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.360355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.360396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-cabundle-cert\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.360434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tls-certs\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.360465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-home\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.360536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkl82\" (UniqueName: \"kubernetes.io/projected/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kube-api-access-qkl82\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.360574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-model-cache\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.361026    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-model-cache\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.363477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.361339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tmp-dir\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.365337 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.365304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.365477 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.365385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-home\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.366310 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.365965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-cabundle-cert\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.370960 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.370761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-dshm\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.373079 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.373046    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tls-certs\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.374410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.374381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkl82\" (UniqueName: \"kubernetes.io/projected/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kube-api-access-qkl82\") pod \"lora-single-adapter-test-kserve-66f7fc75c6-wgwsr\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.494005 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.493910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:33.782276 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:33.782246    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"]
Sep 09 21:53:33.783722 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:53:33.783687    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a379cf_38e7_4e3e_b6ac_b6041203aaa4.slice/crio-cabb7f0e02536d068e1074998a98cadfd88378a21b0a2070240d99dabe5666f8 WatchSource:0}: Error finding container cabb7f0e02536d068e1074998a98cadfd88378a21b0a2070240d99dabe5666f8: Status 404 returned error can't find the container with id cabb7f0e02536d068e1074998a98cadfd88378a21b0a2070240d99dabe5666f8
Sep 09 21:53:34.251037 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:34.250999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" event={"ID":"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4","Type":"ContainerStarted","Data":"40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1"}
Sep 09 21:53:34.251568 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:34.251059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" event={"ID":"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4","Type":"ContainerStarted","Data":"cabb7f0e02536d068e1074998a98cadfd88378a21b0a2070240d99dabe5666f8"}
Sep 09 21:53:37.270444 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:37.270408    2583 generic.go:356] "Generic (PLEG): container finished" podID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerID="40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1" exitCode=0
Sep 09 21:53:37.270911 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:37.270484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" event={"ID":"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4","Type":"ContainerDied","Data":"40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1"}
Sep 09 21:53:38.278952 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:38.278910    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" event={"ID":"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4","Type":"ContainerStarted","Data":"4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404"}
Sep 09 21:53:38.305283 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:38.305206    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podStartSLOduration=2.291875558 podStartE2EDuration="5.305185153s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.013309595s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:53:38.303532281 +0000 UTC m=+3656.928006231" watchObservedRunningTime="2026-09-09 21:53:38.305185153 +0000 UTC m=+3656.929659104"
Sep 09 21:53:38.494767 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:38.494727    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:43.495066 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:43.495023    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:53:43.496785 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:43.496751    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 09 21:53:53.495197 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:53:53.495150    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 09 21:54:03.494630 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:54:03.494528    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 09 21:54:13.495076 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:54:13.495024    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 09 21:54:23.495324 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:54:23.495276    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 09 21:54:33.494776 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:54:33.494715    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 09 21:54:43.494783 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:54:43.494741    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 09 21:54:53.504949 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:54:53.504915    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:54:53.512932 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:54:53.512908    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:55:19.820212 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:19.820177    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-66f7fc75c6-wgwsr_d9a379cf-38e7-4e3e-b6ac-b6041203aaa4/storage-initializer/0.log"
Sep 09 21:55:19.834557 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:19.834532    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-66f7fc75c6-wgwsr_d9a379cf-38e7-4e3e-b6ac-b6041203aaa4/main/0.log"
Sep 09 21:55:21.008194 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:21.008163    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"]
Sep 09 21:55:36.031891 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.031796    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main" containerID="cri-o://4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404" gracePeriod=45
Sep 09 21:55:36.506367 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.506328    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 09 21:55:36.686519 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.686481    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:55:36.809640 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.809611    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-home\") pod \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") "
Sep 09 21:55:36.809830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.809647    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-model-cache\") pod \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") "
Sep 09 21:55:36.809830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.809687    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-cabundle-cert\") pod \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") "
Sep 09 21:55:36.809830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.809703    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tmp-dir\") pod \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") "
Sep 09 21:55:36.809830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.809754    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tls-certs\") pod \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") "
Sep 09 21:55:36.809830 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.809773    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-dshm\") pod \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") "
Sep 09 21:55:36.810154 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.809837    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kserve-provision-location\") pod \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") "
Sep 09 21:55:36.810154 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.809860    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkl82\" (UniqueName: \"kubernetes.io/projected/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kube-api-access-qkl82\") pod \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\" (UID: \"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4\") "
Sep 09 21:55:36.810154 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.810049    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" (UID: "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:36.810154 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.810124    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" (UID: "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:55:36.810367 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.810252    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:55:36.810367 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.810310    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:55:36.810367 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.810243    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-home" (OuterVolumeSpecName: "home") pod "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" (UID: "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:36.812694 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.812659    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kube-api-access-qkl82" (OuterVolumeSpecName: "kube-api-access-qkl82") pod "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" (UID: "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4"). InnerVolumeSpecName "kube-api-access-qkl82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:55:36.812982 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.812956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-dshm" (OuterVolumeSpecName: "dshm") pod "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" (UID: "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:36.813059 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.812956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" (UID: "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:55:36.822232 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.822205    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" (UID: "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:36.833815 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.833776    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" (UID: "d9a379cf-38e7-4e3e-b6ac-b6041203aaa4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:36.901370 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.901329    2583 generic.go:356] "Generic (PLEG): container finished" podID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerID="4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404" exitCode=0
Sep 09 21:55:36.901505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.901374    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"
Sep 09 21:55:36.901505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.901373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" event={"ID":"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4","Type":"ContainerDied","Data":"4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404"}
Sep 09 21:55:36.901505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.901423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr" event={"ID":"d9a379cf-38e7-4e3e-b6ac-b6041203aaa4","Type":"ContainerDied","Data":"cabb7f0e02536d068e1074998a98cadfd88378a21b0a2070240d99dabe5666f8"}
Sep 09 21:55:36.901505 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.901444    2583 scope.go:117] "RemoveContainer" containerID="4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404"
Sep 09 21:55:36.910603 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.910583    2583 scope.go:117] "RemoveContainer" containerID="40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1"
Sep 09 21:55:36.910865 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.910844    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:55:36.910921 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.910870    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:55:36.910921 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.910885    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:55:36.910921 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.910896    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:55:36.910921 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.910905    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:55:36.910921 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.910915    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkl82\" (UniqueName: \"kubernetes.io/projected/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4-kube-api-access-qkl82\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:55:36.918693 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.918678    2583 scope.go:117] "RemoveContainer" containerID="4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404"
Sep 09 21:55:36.918963 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:55:36.918944    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404\": container with ID starting with 4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404 not found: ID does not exist" containerID="4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404"
Sep 09 21:55:36.919008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.918971    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404"} err="failed to get container status \"4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404\": rpc error: code = NotFound desc = could not find container \"4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404\": container with ID starting with 4aa67f86afa977ce5379cc063ec717f50a8df9a67ad5cac1a21b1728295f0404 not found: ID does not exist"
Sep 09 21:55:36.919008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.918986    2583 scope.go:117] "RemoveContainer" containerID="40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1"
Sep 09 21:55:36.919238 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:55:36.919221    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1\": container with ID starting with 40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1 not found: ID does not exist" containerID="40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1"
Sep 09 21:55:36.919281 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.919243    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1"} err="failed to get container status \"40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1\": rpc error: code = NotFound desc = could not find container \"40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1\": container with ID starting with 40120d748a2ca0a66b1e4f8374dabe6e29d6335d31becbd9e848562c73c418d1 not found: ID does not exist"
Sep 09 21:55:36.924513 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.924490    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"]
Sep 09 21:55:36.927833 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:36.927809    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-66f7fc75c6-wgwsr"]
Sep 09 21:55:37.866805 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:37.866772    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" path="/var/lib/kubelet/pods/d9a379cf-38e7-4e3e-b6ac-b6041203aaa4/volumes"
Sep 09 21:55:54.096037 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.096000    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"]
Sep 09 21:55:54.096856 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.096833    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main"
Sep 09 21:55:54.096856 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.096864    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main"
Sep 09 21:55:54.097025 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.096906    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="storage-initializer"
Sep 09 21:55:54.097025 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.096917    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="storage-initializer"
Sep 09 21:55:54.097156 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.097080    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a379cf-38e7-4e3e-b6ac-b6041203aaa4" containerName="main"
Sep 09 21:55:54.103151 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.103126    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.105883 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.105864    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 09 21:55:54.106021 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.105958    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:55:54.106021 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.105963    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-4szqd\""
Sep 09 21:55:54.106157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.106139    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 09 21:55:54.106255 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.106238    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 09 21:55:54.106337 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.106319    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 09 21:55:54.115196 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.115172    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"]
Sep 09 21:55:54.249406 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.249368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tls-certs\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.249575 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.249417    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5kg8\" (UniqueName: \"kubernetes.io/projected/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kube-api-access-l5kg8\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.249575 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.249447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.249575 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.249478    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-home\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.249575 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.249516    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.249575 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.249555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff634fd4-cf8c-4408-8b0a-3ae98e234943-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.249741 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.249636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-model-cache\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.249741 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.249657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-dshm\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350153 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350153 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff634fd4-cf8c-4408-8b0a-3ae98e234943-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350153 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-model-cache\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350511 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-dshm\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350511 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tls-certs\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350511 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5kg8\" (UniqueName: \"kubernetes.io/projected/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kube-api-access-l5kg8\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350511 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350511 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-home\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350799 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.350799 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-home\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.351018 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.350996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.351133 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.351089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff634fd4-cf8c-4408-8b0a-3ae98e234943-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.351217 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.351193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-model-cache\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.352938 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.352914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-dshm\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.353037 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.353002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tls-certs\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.362236 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.362202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5kg8\" (UniqueName: \"kubernetes.io/projected/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kube-api-access-l5kg8\") pod \"lora-multiple-adapters-test-kserve-bc97b7d85-h78km\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.415734 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.415689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:55:54.703938 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.703912    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"]
Sep 09 21:55:54.706008 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:55:54.705982    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff634fd4_cf8c_4408_8b0a_3ae98e234943.slice/crio-f11ed203223753d0e19292e1d9e6f4b601771048d39d656cd43ca6fd083ac968 WatchSource:0}: Error finding container f11ed203223753d0e19292e1d9e6f4b601771048d39d656cd43ca6fd083ac968: Status 404 returned error can't find the container with id f11ed203223753d0e19292e1d9e6f4b601771048d39d656cd43ca6fd083ac968
Sep 09 21:55:54.986423 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.986319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" event={"ID":"ff634fd4-cf8c-4408-8b0a-3ae98e234943","Type":"ContainerStarted","Data":"78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6"}
Sep 09 21:55:54.986423 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:54.986379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" event={"ID":"ff634fd4-cf8c-4408-8b0a-3ae98e234943","Type":"ContainerStarted","Data":"f11ed203223753d0e19292e1d9e6f4b601771048d39d656cd43ca6fd083ac968"}
Sep 09 21:55:58.002988 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:58.002951    2583 generic.go:356] "Generic (PLEG): container finished" podID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerID="78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6" exitCode=0
Sep 09 21:55:58.003379 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:58.003025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" event={"ID":"ff634fd4-cf8c-4408-8b0a-3ae98e234943","Type":"ContainerDied","Data":"78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6"}
Sep 09 21:55:59.012203 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:59.012168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" event={"ID":"ff634fd4-cf8c-4408-8b0a-3ae98e234943","Type":"ContainerStarted","Data":"9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980"}
Sep 09 21:55:59.041410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:59.041354    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podStartSLOduration=1.89684514 podStartE2EDuration="5.041336767s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.144491627s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:59.038745889 +0000 UTC m=+3797.663219839" watchObservedRunningTime="2026-09-09 21:55:59.041336767 +0000 UTC m=+3797.665810708"
Sep 09 21:55:59.416220 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:55:59.416187    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:56:04.416652 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:56:04.416607    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:56:04.418271 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:56:04.418239    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 09 21:56:14.416310 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:56:14.416264    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 09 21:56:24.416678 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:56:24.416637    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 09 21:56:34.416367 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:56:34.416316    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 09 21:56:44.416351 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:56:44.416299    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 09 21:56:54.416467 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:56:54.416418    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 09 21:57:04.417157 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:04.417071    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 09 21:57:14.427567 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:14.427524    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:57:14.436027 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:14.435997    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:57:42.374493 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:42.374458    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-bc97b7d85-h78km_ff634fd4-cf8c-4408-8b0a-3ae98e234943/storage-initializer/0.log"
Sep 09 21:57:42.385849 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:42.385817    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-bc97b7d85-h78km_ff634fd4-cf8c-4408-8b0a-3ae98e234943/main/0.log"
Sep 09 21:57:43.894141 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:43.894095    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"]
Sep 09 21:57:58.917161 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:58.917060    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main" containerID="cri-o://9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980" gracePeriod=45
Sep 09 21:57:59.428462 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.428419    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 09 21:57:59.563599 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.563571    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:57:59.641914 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.641809    2583 generic.go:356] "Generic (PLEG): container finished" podID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerID="9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980" exitCode=0
Sep 09 21:57:59.641914 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.641860    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"
Sep 09 21:57:59.641914 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.641867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" event={"ID":"ff634fd4-cf8c-4408-8b0a-3ae98e234943","Type":"ContainerDied","Data":"9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980"}
Sep 09 21:57:59.642209 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.641933    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km" event={"ID":"ff634fd4-cf8c-4408-8b0a-3ae98e234943","Type":"ContainerDied","Data":"f11ed203223753d0e19292e1d9e6f4b601771048d39d656cd43ca6fd083ac968"}
Sep 09 21:57:59.642209 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.641957    2583 scope.go:117] "RemoveContainer" containerID="9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980"
Sep 09 21:57:59.650636 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.650614    2583 scope.go:117] "RemoveContainer" containerID="78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6"
Sep 09 21:57:59.661013 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.660988    2583 scope.go:117] "RemoveContainer" containerID="9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980"
Sep 09 21:57:59.661521 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:57:59.661500    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980\": container with ID starting with 9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980 not found: ID does not exist" containerID="9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980"
Sep 09 21:57:59.661568 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.661532    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980"} err="failed to get container status \"9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980\": rpc error: code = NotFound desc = could not find container \"9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980\": container with ID starting with 9d9d01363388f1f972889ee4e71c29b514ee7a4d65c34dd741aad47f877ee980 not found: ID does not exist"
Sep 09 21:57:59.661568 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.661552    2583 scope.go:117] "RemoveContainer" containerID="78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6"
Sep 09 21:57:59.661805 ip-10-0-140-132 kubenswrapper[2583]: E0909 21:57:59.661777    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6\": container with ID starting with 78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6 not found: ID does not exist" containerID="78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6"
Sep 09 21:57:59.661845 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.661814    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6"} err="failed to get container status \"78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6\": rpc error: code = NotFound desc = could not find container \"78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6\": container with ID starting with 78ee25ccbe291042d4744ed9b39f9882094fbc800557438f946d71ac1d41d5c6 not found: ID does not exist"
Sep 09 21:57:59.724355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724317    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tls-certs\") pod \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") "
Sep 09 21:57:59.724355 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724360    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-dshm\") pod \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") "
Sep 09 21:57:59.724579 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-home\") pod \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") "
Sep 09 21:57:59.724579 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724425    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kserve-provision-location\") pod \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") "
Sep 09 21:57:59.724579 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724488    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-model-cache\") pod \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") "
Sep 09 21:57:59.724579 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724511    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5kg8\" (UniqueName: \"kubernetes.io/projected/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kube-api-access-l5kg8\") pod \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") "
Sep 09 21:57:59.724579 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724533    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tmp-dir\") pod \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") "
Sep 09 21:57:59.725036 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724582    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff634fd4-cf8c-4408-8b0a-3ae98e234943-cabundle-cert\") pod \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\" (UID: \"ff634fd4-cf8c-4408-8b0a-3ae98e234943\") "
Sep 09 21:57:59.725036 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724760    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff634fd4-cf8c-4408-8b0a-3ae98e234943" (UID: "ff634fd4-cf8c-4408-8b0a-3ae98e234943"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:57:59.725036 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.724892    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-home" (OuterVolumeSpecName: "home") pod "ff634fd4-cf8c-4408-8b0a-3ae98e234943" (UID: "ff634fd4-cf8c-4408-8b0a-3ae98e234943"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:57:59.725275 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.725001    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:57:59.725275 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.725085    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:57:59.725402 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.725377    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff634fd4-cf8c-4408-8b0a-3ae98e234943-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff634fd4-cf8c-4408-8b0a-3ae98e234943" (UID: "ff634fd4-cf8c-4408-8b0a-3ae98e234943"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:57:59.727026 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.726994    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-dshm" (OuterVolumeSpecName: "dshm") pod "ff634fd4-cf8c-4408-8b0a-3ae98e234943" (UID: "ff634fd4-cf8c-4408-8b0a-3ae98e234943"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:57:59.727180 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.727045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff634fd4-cf8c-4408-8b0a-3ae98e234943" (UID: "ff634fd4-cf8c-4408-8b0a-3ae98e234943"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:57:59.727495 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.727468    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kube-api-access-l5kg8" (OuterVolumeSpecName: "kube-api-access-l5kg8") pod "ff634fd4-cf8c-4408-8b0a-3ae98e234943" (UID: "ff634fd4-cf8c-4408-8b0a-3ae98e234943"). InnerVolumeSpecName "kube-api-access-l5kg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:57:59.736796 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.736762    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff634fd4-cf8c-4408-8b0a-3ae98e234943" (UID: "ff634fd4-cf8c-4408-8b0a-3ae98e234943"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:57:59.749940 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.749896    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff634fd4-cf8c-4408-8b0a-3ae98e234943" (UID: "ff634fd4-cf8c-4408-8b0a-3ae98e234943"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:57:59.826283 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.826241    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff634fd4-cf8c-4408-8b0a-3ae98e234943-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:57:59.826283 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.826274    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:57:59.826283 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.826285    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:57:59.826283 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.826294    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:57:59.826547 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.826303    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5kg8\" (UniqueName: \"kubernetes.io/projected/ff634fd4-cf8c-4408-8b0a-3ae98e234943-kube-api-access-l5kg8\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:57:59.826547 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.826312    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff634fd4-cf8c-4408-8b0a-3ae98e234943-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 21:57:59.967614 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.967523    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"]
Sep 09 21:57:59.973969 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:57:59.973942    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bc97b7d85-h78km"]
Sep 09 21:58:01.865313 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:01.865279    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" path="/var/lib/kubelet/pods/ff634fd4-cf8c-4408-8b0a-3ae98e234943/volumes"
Sep 09 21:58:23.009537 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.009501    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"]
Sep 09 21:58:23.010008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.009854    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main"
Sep 09 21:58:23.010008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.009866    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main"
Sep 09 21:58:23.010008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.009888    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="storage-initializer"
Sep 09 21:58:23.010008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.009893    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="storage-initializer"
Sep 09 21:58:23.010008 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.009951    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff634fd4-cf8c-4408-8b0a-3ae98e234943" containerName="main"
Sep 09 21:58:23.014680 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.014658    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.017580 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.017556    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 09 21:58:23.017709 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.017680    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:58:23.017814 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.017797    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 09 21:58:23.017887 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.017845    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 09 21:58:23.017943 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.017911    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-8wjff\""
Sep 09 21:58:23.017943 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.017913    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 09 21:58:23.029410 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.029379    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"]
Sep 09 21:58:23.143088 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.143048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kserve-provision-location\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.143287 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.143097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-model-cache\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.143287 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.143203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-cabundle-cert\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.143287 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.143234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqlqg\" (UniqueName: \"kubernetes.io/projected/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kube-api-access-mqlqg\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.143287 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.143256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-home\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.143287 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.143274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tmp-dir\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.143461 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.143317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tls-certs\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.143461 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.143396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-dshm\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.244514 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-dshm\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.244681 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kserve-provision-location\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.244681 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-model-cache\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.244769 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-cabundle-cert\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.244769 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqlqg\" (UniqueName: \"kubernetes.io/projected/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kube-api-access-mqlqg\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.244769 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-home\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.244769 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tmp-dir\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.244956 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tls-certs\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.245018 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kserve-provision-location\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.245018 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.244987    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-model-cache\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.245153 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.245134    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tmp-dir\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.245225 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.245208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-home\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.245501 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.245480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-cabundle-cert\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.247175 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.247154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-dshm\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.247516 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.247500    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tls-certs\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.260943 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.260874    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqlqg\" (UniqueName: \"kubernetes.io/projected/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kube-api-access-mqlqg\") pod \"tls-verification-test-kserve-7b4489c6d5-2ldj8\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.325627 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.325589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:23.614841 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.614683    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"]
Sep 09 21:58:23.617685 ip-10-0-140-132 kubenswrapper[2583]: W0909 21:58:23.617660    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db3c6b8_a767_461f_8a31_e4c0eccac5fa.slice/crio-8df2553539637b5810ed2bbe2660679e8500ef0ea61e0e83be0431cb0e4b3d46 WatchSource:0}: Error finding container 8df2553539637b5810ed2bbe2660679e8500ef0ea61e0e83be0431cb0e4b3d46: Status 404 returned error can't find the container with id 8df2553539637b5810ed2bbe2660679e8500ef0ea61e0e83be0431cb0e4b3d46
Sep 09 21:58:23.755324 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.755285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" event={"ID":"1db3c6b8-a767-461f-8a31-e4c0eccac5fa","Type":"ContainerStarted","Data":"21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a"}
Sep 09 21:58:23.755510 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:23.755346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" event={"ID":"1db3c6b8-a767-461f-8a31-e4c0eccac5fa","Type":"ContainerStarted","Data":"8df2553539637b5810ed2bbe2660679e8500ef0ea61e0e83be0431cb0e4b3d46"}
Sep 09 21:58:26.774022 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:26.773986    2583 generic.go:356] "Generic (PLEG): container finished" podID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerID="21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a" exitCode=0
Sep 09 21:58:26.774456 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:26.774061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" event={"ID":"1db3c6b8-a767-461f-8a31-e4c0eccac5fa","Type":"ContainerDied","Data":"21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a"}
Sep 09 21:58:27.781419 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:27.781380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" event={"ID":"1db3c6b8-a767-461f-8a31-e4c0eccac5fa","Type":"ContainerStarted","Data":"686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4"}
Sep 09 21:58:27.808450 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:27.808396    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podStartSLOduration=3.560664665 podStartE2EDuration="5.808378086s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.247713421s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:58:27.805604695 +0000 UTC m=+3946.430078636" watchObservedRunningTime="2026-09-09 21:58:27.808378086 +0000 UTC m=+3946.432852027"
Sep 09 21:58:28.326229 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:28.326189    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:33.328013 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:33.327891    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 21:58:33.329475 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:33.329434    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 21:58:43.325981 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:43.325935    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 21:58:53.326731 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:58:53.326686    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 21:59:03.326629 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:59:03.326582    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 21:59:13.326694 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:59:13.326644    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 21:59:23.326212 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:59:23.326168    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 21:59:33.326071 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:59:33.326018    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 21:59:43.326328 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:59:43.326277    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 21:59:53.326286 ip-10-0-140-132 kubenswrapper[2583]: I0909 21:59:53.326241    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 22:00:03.326947 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:03.326851    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 22:00:13.326521 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:13.326472    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 22:00:23.336497 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:23.336466    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 22:00:23.344381 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:23.344345    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 22:00:28.150322 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:28.150290    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"]
Sep 09 22:00:43.177318 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.177241    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" containerID="cri-o://686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4" gracePeriod=45
Sep 09 22:00:43.337873 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.337538    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 09 22:00:43.938281 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.938257    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 22:00:43.981029 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.980923    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kserve-provision-location\") pod \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") "
Sep 09 22:00:43.981029 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.980983    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-dshm\") pod \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") "
Sep 09 22:00:43.981029 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.981013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-home\") pod \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") "
Sep 09 22:00:43.981533 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.981041    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tls-certs\") pod \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") "
Sep 09 22:00:43.981533 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.981082    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqlqg\" (UniqueName: \"kubernetes.io/projected/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kube-api-access-mqlqg\") pod \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") "
Sep 09 22:00:43.981533 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.981140    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tmp-dir\") pod \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") "
Sep 09 22:00:43.981533 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.981213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-model-cache\") pod \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") "
Sep 09 22:00:43.981533 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.981235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-cabundle-cert\") pod \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\" (UID: \"1db3c6b8-a767-461f-8a31-e4c0eccac5fa\") "
Sep 09 22:00:43.982260 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.982055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-home" (OuterVolumeSpecName: "home") pod "1db3c6b8-a767-461f-8a31-e4c0eccac5fa" (UID: "1db3c6b8-a767-461f-8a31-e4c0eccac5fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:43.982419 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.982377    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "1db3c6b8-a767-461f-8a31-e4c0eccac5fa" (UID: "1db3c6b8-a767-461f-8a31-e4c0eccac5fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:43.982490 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.982409    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1db3c6b8-a767-461f-8a31-e4c0eccac5fa" (UID: "1db3c6b8-a767-461f-8a31-e4c0eccac5fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:00:43.984818 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.984792    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-dshm" (OuterVolumeSpecName: "dshm") pod "1db3c6b8-a767-461f-8a31-e4c0eccac5fa" (UID: "1db3c6b8-a767-461f-8a31-e4c0eccac5fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:43.984928 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.984862    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kube-api-access-mqlqg" (OuterVolumeSpecName: "kube-api-access-mqlqg") pod "1db3c6b8-a767-461f-8a31-e4c0eccac5fa" (UID: "1db3c6b8-a767-461f-8a31-e4c0eccac5fa"). InnerVolumeSpecName "kube-api-access-mqlqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:43.985336 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.985305    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1db3c6b8-a767-461f-8a31-e4c0eccac5fa" (UID: "1db3c6b8-a767-461f-8a31-e4c0eccac5fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:00:43.996284 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:43.996247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1db3c6b8-a767-461f-8a31-e4c0eccac5fa" (UID: "1db3c6b8-a767-461f-8a31-e4c0eccac5fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:44.013264 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.013215    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1db3c6b8-a767-461f-8a31-e4c0eccac5fa" (UID: "1db3c6b8-a767-461f-8a31-e4c0eccac5fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:44.082494 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.082459    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:00:44.082655 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.082503    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:00:44.082655 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.082513    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:00:44.082655 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.082525    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqlqg\" (UniqueName: \"kubernetes.io/projected/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kube-api-access-mqlqg\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:00:44.082655 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.082550    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:00:44.082655 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.082562    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:00:44.082655 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.082573    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:00:44.082655 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.082581    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db3c6b8-a767-461f-8a31-e4c0eccac5fa-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:00:44.491841 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.491802    2583 generic.go:356] "Generic (PLEG): container finished" podID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerID="686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4" exitCode=0
Sep 09 22:00:44.492353 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.491850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" event={"ID":"1db3c6b8-a767-461f-8a31-e4c0eccac5fa","Type":"ContainerDied","Data":"686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4"}
Sep 09 22:00:44.492353 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.491862    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"
Sep 09 22:00:44.492353 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.491906    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8" event={"ID":"1db3c6b8-a767-461f-8a31-e4c0eccac5fa","Type":"ContainerDied","Data":"8df2553539637b5810ed2bbe2660679e8500ef0ea61e0e83be0431cb0e4b3d46"}
Sep 09 22:00:44.492353 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.491937    2583 scope.go:117] "RemoveContainer" containerID="686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4"
Sep 09 22:00:44.501802 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.501780    2583 scope.go:117] "RemoveContainer" containerID="21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a"
Sep 09 22:00:44.510235 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.510215    2583 scope.go:117] "RemoveContainer" containerID="686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4"
Sep 09 22:00:44.510536 ip-10-0-140-132 kubenswrapper[2583]: E0909 22:00:44.510519    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4\": container with ID starting with 686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4 not found: ID does not exist" containerID="686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4"
Sep 09 22:00:44.510594 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.510545    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4"} err="failed to get container status \"686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4\": rpc error: code = NotFound desc = could not find container \"686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4\": container with ID starting with 686602ac49760ce4d55e49781726cc1025cee2898d1307ff35ff0e42899beec4 not found: ID does not exist"
Sep 09 22:00:44.510594 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.510561    2583 scope.go:117] "RemoveContainer" containerID="21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a"
Sep 09 22:00:44.510851 ip-10-0-140-132 kubenswrapper[2583]: E0909 22:00:44.510831    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a\": container with ID starting with 21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a not found: ID does not exist" containerID="21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a"
Sep 09 22:00:44.510904 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.510856    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a"} err="failed to get container status \"21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a\": rpc error: code = NotFound desc = could not find container \"21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a\": container with ID starting with 21f86e6f61bab700de82bf8d44907031ae5103f60052114f3c0c92f22bf2a55a not found: ID does not exist"
Sep 09 22:00:44.515677 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.515650    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"]
Sep 09 22:00:44.521410 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:44.521376    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b4489c6d5-2ldj8"]
Sep 09 22:00:45.062767 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.062722    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"]
Sep 09 22:00:45.063226 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.063206    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main"
Sep 09 22:00:45.063226 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.063227    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main"
Sep 09 22:00:45.063399 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.063249    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="storage-initializer"
Sep 09 22:00:45.063399 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.063255    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="storage-initializer"
Sep 09 22:00:45.063399 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.063319    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" containerName="main"
Sep 09 22:00:45.068013 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.067983    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.071638 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.071612    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-6xgvz\""
Sep 09 22:00:45.071783 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.071763    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 09 22:00:45.072052 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.072029    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 22:00:45.072336 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.072310    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 09 22:00:45.072930 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.072863    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 09 22:00:45.073050 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.073033    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 09 22:00:45.079776 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.079750    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"]
Sep 09 22:00:45.091855 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.091815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.091855 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.091857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-home\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.092135 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.091874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.092135 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.091913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac05f5e0-7f37-478c-810a-23660cc65cf3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.092135 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.091930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9n2\" (UniqueName: \"kubernetes.io/projected/ac05f5e0-7f37-478c-810a-23660cc65cf3-kube-api-access-9r9n2\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.092135 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.091950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.092135 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.091967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac05f5e0-7f37-478c-810a-23660cc65cf3-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.092135 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.092076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.193478 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.193669 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-home\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.193669 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.193669 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac05f5e0-7f37-478c-810a-23660cc65cf3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.193669 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193569    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9n2\" (UniqueName: \"kubernetes.io/projected/ac05f5e0-7f37-478c-810a-23660cc65cf3-kube-api-access-9r9n2\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.193669 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.193927 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193759    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac05f5e0-7f37-478c-810a-23660cc65cf3-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.193927 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.194037 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193952    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.194037 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.193999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.194037 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.194001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.194271 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.194250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-home\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.194551 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.194532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac05f5e0-7f37-478c-810a-23660cc65cf3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.196404 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.196375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.196488 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.196427    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac05f5e0-7f37-478c-810a-23660cc65cf3-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.203420 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.203390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9n2\" (UniqueName: \"kubernetes.io/projected/ac05f5e0-7f37-478c-810a-23660cc65cf3-kube-api-access-9r9n2\") pod \"router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.380787 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.380696    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:45.665920 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.665894    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"]
Sep 09 22:00:45.667849 ip-10-0-140-132 kubenswrapper[2583]: W0909 22:00:45.667820    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac05f5e0_7f37_478c_810a_23660cc65cf3.slice/crio-64d206c82b5fb6f43ed7a07ac9fa3bd1becf91b3985a0e440e6ecd1abdb97ce8 WatchSource:0}: Error finding container 64d206c82b5fb6f43ed7a07ac9fa3bd1becf91b3985a0e440e6ecd1abdb97ce8: Status 404 returned error can't find the container with id 64d206c82b5fb6f43ed7a07ac9fa3bd1becf91b3985a0e440e6ecd1abdb97ce8
Sep 09 22:00:45.866672 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:45.866637    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1db3c6b8-a767-461f-8a31-e4c0eccac5fa" path="/var/lib/kubelet/pods/1db3c6b8-a767-461f-8a31-e4c0eccac5fa/volumes"
Sep 09 22:00:46.502475 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:46.502442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" event={"ID":"ac05f5e0-7f37-478c-810a-23660cc65cf3","Type":"ContainerStarted","Data":"f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e"}
Sep 09 22:00:46.502659 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:46.502493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" event={"ID":"ac05f5e0-7f37-478c-810a-23660cc65cf3","Type":"ContainerStarted","Data":"64d206c82b5fb6f43ed7a07ac9fa3bd1becf91b3985a0e440e6ecd1abdb97ce8"}
Sep 09 22:00:48.512726 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:48.512693    2583 generic.go:356] "Generic (PLEG): container finished" podID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerID="f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e" exitCode=0
Sep 09 22:00:48.513174 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:48.512767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" event={"ID":"ac05f5e0-7f37-478c-810a-23660cc65cf3","Type":"ContainerDied","Data":"f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e"}
Sep 09 22:00:49.520645 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:49.520611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" event={"ID":"ac05f5e0-7f37-478c-810a-23660cc65cf3","Type":"ContainerStarted","Data":"c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e"}
Sep 09 22:00:49.545218 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:49.545159    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podStartSLOduration=2.315010092 podStartE2EDuration="4.545142182s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.23013209s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:49.543515179 +0000 UTC m=+4088.167989121" watchObservedRunningTime="2026-09-09 22:00:49.545142182 +0000 UTC m=+4088.169616123"
Sep 09 22:00:55.381298 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:55.381251    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:55.381707 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:55.381325    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:00:55.383054 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:00:55.383025    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:01:05.381483 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:01:05.381441    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:01:15.381524 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:01:15.381475    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:01:25.381292 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:01:25.381246    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:01:35.382288 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:01:35.382183    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:01:45.382023 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:01:45.381951    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:01:55.381316 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:01:55.381272    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:02:05.382083 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:02:05.382038    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:02:15.381350 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:02:15.381301    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:02:25.381220 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:02:25.381174    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:02:35.381438 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:02:35.381389    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 09 22:02:45.391752 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:02:45.391716    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:02:45.400174 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:02:45.400149    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:02:57.941555 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:02:57.941513    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"]
Sep 09 22:03:12.965530 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:12.965467    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main" containerID="cri-o://c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e" gracePeriod=45
Sep 09 22:03:13.719975 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.719948    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:03:13.864514 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.864481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r9n2\" (UniqueName: \"kubernetes.io/projected/ac05f5e0-7f37-478c-810a-23660cc65cf3-kube-api-access-9r9n2\") pod \"ac05f5e0-7f37-478c-810a-23660cc65cf3\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") "
Sep 09 22:03:13.864514 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.864513    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-tmp-dir\") pod \"ac05f5e0-7f37-478c-810a-23660cc65cf3\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") "
Sep 09 22:03:13.864770 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.864544    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac05f5e0-7f37-478c-810a-23660cc65cf3-cabundle-cert\") pod \"ac05f5e0-7f37-478c-810a-23660cc65cf3\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") "
Sep 09 22:03:13.864770 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.864573    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-kserve-provision-location\") pod \"ac05f5e0-7f37-478c-810a-23660cc65cf3\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") "
Sep 09 22:03:13.864770 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.864595    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac05f5e0-7f37-478c-810a-23660cc65cf3-tls-certs\") pod \"ac05f5e0-7f37-478c-810a-23660cc65cf3\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") "
Sep 09 22:03:13.864770 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.864676    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-model-cache\") pod \"ac05f5e0-7f37-478c-810a-23660cc65cf3\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") "
Sep 09 22:03:13.864770 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.864699    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-dshm\") pod \"ac05f5e0-7f37-478c-810a-23660cc65cf3\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") "
Sep 09 22:03:13.864770 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.864737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-home\") pod \"ac05f5e0-7f37-478c-810a-23660cc65cf3\" (UID: \"ac05f5e0-7f37-478c-810a-23660cc65cf3\") "
Sep 09 22:03:13.865089 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.864969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac05f5e0-7f37-478c-810a-23660cc65cf3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ac05f5e0-7f37-478c-810a-23660cc65cf3" (UID: "ac05f5e0-7f37-478c-810a-23660cc65cf3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:03:13.865089 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.865070    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac05f5e0-7f37-478c-810a-23660cc65cf3-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:03:13.865474 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.865445    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac05f5e0-7f37-478c-810a-23660cc65cf3" (UID: "ac05f5e0-7f37-478c-810a-23660cc65cf3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:13.865630 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.865605    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-home" (OuterVolumeSpecName: "home") pod "ac05f5e0-7f37-478c-810a-23660cc65cf3" (UID: "ac05f5e0-7f37-478c-810a-23660cc65cf3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:13.867372 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.867343    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac05f5e0-7f37-478c-810a-23660cc65cf3-kube-api-access-9r9n2" (OuterVolumeSpecName: "kube-api-access-9r9n2") pod "ac05f5e0-7f37-478c-810a-23660cc65cf3" (UID: "ac05f5e0-7f37-478c-810a-23660cc65cf3"). InnerVolumeSpecName "kube-api-access-9r9n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:13.867692 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.867658    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac05f5e0-7f37-478c-810a-23660cc65cf3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac05f5e0-7f37-478c-810a-23660cc65cf3" (UID: "ac05f5e0-7f37-478c-810a-23660cc65cf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:03:13.867692 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.867681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-dshm" (OuterVolumeSpecName: "dshm") pod "ac05f5e0-7f37-478c-810a-23660cc65cf3" (UID: "ac05f5e0-7f37-478c-810a-23660cc65cf3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:13.878257 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.878230    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac05f5e0-7f37-478c-810a-23660cc65cf3" (UID: "ac05f5e0-7f37-478c-810a-23660cc65cf3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:13.892267 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.892237    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac05f5e0-7f37-478c-810a-23660cc65cf3" (UID: "ac05f5e0-7f37-478c-810a-23660cc65cf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:13.966488 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.966457    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:03:13.966488 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.966486    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac05f5e0-7f37-478c-810a-23660cc65cf3-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:03:13.966488 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.966495    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:03:13.966488 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.966506    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:03:13.966959 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.966517    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:03:13.966959 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.966528    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r9n2\" (UniqueName: \"kubernetes.io/projected/ac05f5e0-7f37-478c-810a-23660cc65cf3-kube-api-access-9r9n2\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:03:13.966959 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:13.966539    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac05f5e0-7f37-478c-810a-23660cc65cf3-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:03:14.259000 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.258908    2583 generic.go:356] "Generic (PLEG): container finished" podID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerID="c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e" exitCode=0
Sep 09 22:03:14.259000 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.258957    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"
Sep 09 22:03:14.259229 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.258960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" event={"ID":"ac05f5e0-7f37-478c-810a-23660cc65cf3","Type":"ContainerDied","Data":"c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e"}
Sep 09 22:03:14.259229 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.259087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6" event={"ID":"ac05f5e0-7f37-478c-810a-23660cc65cf3","Type":"ContainerDied","Data":"64d206c82b5fb6f43ed7a07ac9fa3bd1becf91b3985a0e440e6ecd1abdb97ce8"}
Sep 09 22:03:14.259229 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.259155    2583 scope.go:117] "RemoveContainer" containerID="c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e"
Sep 09 22:03:14.268691 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.268665    2583 scope.go:117] "RemoveContainer" containerID="f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e"
Sep 09 22:03:14.277176 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.277155    2583 scope.go:117] "RemoveContainer" containerID="c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e"
Sep 09 22:03:14.277468 ip-10-0-140-132 kubenswrapper[2583]: E0909 22:03:14.277449    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e\": container with ID starting with c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e not found: ID does not exist" containerID="c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e"
Sep 09 22:03:14.277522 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.277478    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e"} err="failed to get container status \"c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e\": rpc error: code = NotFound desc = could not find container \"c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e\": container with ID starting with c8f4fc425b70982dc1b8bb7e7e8cda5257c30d009659f4610a2c96f758e7366e not found: ID does not exist"
Sep 09 22:03:14.277522 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.277499    2583 scope.go:117] "RemoveContainer" containerID="f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e"
Sep 09 22:03:14.277772 ip-10-0-140-132 kubenswrapper[2583]: E0909 22:03:14.277751    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e\": container with ID starting with f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e not found: ID does not exist" containerID="f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e"
Sep 09 22:03:14.277814 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.277778    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e"} err="failed to get container status \"f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e\": rpc error: code = NotFound desc = could not find container \"f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e\": container with ID starting with f9ef1abf67115e25d914deea0b5e040584b2bb39951923d1dd1f99b4925c1a2e not found: ID does not exist"
Sep 09 22:03:14.285879 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.285854    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"]
Sep 09 22:03:14.292205 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:14.292182    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8bc85c9cc-vs6h6"]
Sep 09 22:03:15.865659 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:15.865627    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" path="/var/lib/kubelet/pods/ac05f5e0-7f37-478c-810a-23660cc65cf3/volumes"
Sep 09 22:03:34.093901 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.093863    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"]
Sep 09 22:03:34.094327 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.094293    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main"
Sep 09 22:03:34.094327 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.094306    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main"
Sep 09 22:03:34.094401 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.094333    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="storage-initializer"
Sep 09 22:03:34.094401 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.094343    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="storage-initializer"
Sep 09 22:03:34.094468 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.094415    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac05f5e0-7f37-478c-810a-23660cc65cf3" containerName="main"
Sep 09 22:03:34.097492 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.097476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.100272 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.100245    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 09 22:03:34.100430 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.100262    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 22:03:34.100430 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.100245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 09 22:03:34.100430 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.100335    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-nhv94\""
Sep 09 22:03:34.100430 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.100396    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 09 22:03:34.100617 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.100575    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 09 22:03:34.109892 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.109869    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"]
Sep 09 22:03:34.249820 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.249785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-tmp-dir\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.249820 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.249822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmrth\" (UniqueName: \"kubernetes.io/projected/c16a3230-f960-49b8-b922-20e60bd0cdff-kube-api-access-cmrth\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.250035 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.249847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c16a3230-f960-49b8-b922-20e60bd0cdff-tls-certs\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.250035 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.249950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c16a3230-f960-49b8-b922-20e60bd0cdff-cabundle-cert\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.250035 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.249985    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-model-cache\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.250035 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.250006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.250212 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.250051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-home\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.250212 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.250143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-dshm\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.350878 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.350794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-dshm\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.350878 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.350840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-tmp-dir\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.350878 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.350869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrth\" (UniqueName: \"kubernetes.io/projected/c16a3230-f960-49b8-b922-20e60bd0cdff-kube-api-access-cmrth\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351189 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.350900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c16a3230-f960-49b8-b922-20e60bd0cdff-tls-certs\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351189 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.351145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c16a3230-f960-49b8-b922-20e60bd0cdff-cabundle-cert\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351297 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.351200    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-model-cache\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351297 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.351229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351297 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.351271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-home\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351653 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.351595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-model-cache\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351653 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.351642    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351839 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.351653    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-home\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351839 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.351756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-tmp-dir\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.351966 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.351887    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c16a3230-f960-49b8-b922-20e60bd0cdff-cabundle-cert\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.354018 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.353996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-dshm\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.354411 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.354391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c16a3230-f960-49b8-b922-20e60bd0cdff-tls-certs\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.361377 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.361349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmrth\" (UniqueName: \"kubernetes.io/projected/c16a3230-f960-49b8-b922-20e60bd0cdff-kube-api-access-cmrth\") pod \"prestop-hook-test-kserve-7f95b98795-vf8x6\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.408945 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.408899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:34.697378 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:34.697281    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"]
Sep 09 22:03:34.700605 ip-10-0-140-132 kubenswrapper[2583]: W0909 22:03:34.700578    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16a3230_f960_49b8_b922_20e60bd0cdff.slice/crio-44b7e4f03b2d38c887164ca75f8b726ad5ef17117b73bbd7c24f985b707cc0ff WatchSource:0}: Error finding container 44b7e4f03b2d38c887164ca75f8b726ad5ef17117b73bbd7c24f985b707cc0ff: Status 404 returned error can't find the container with id 44b7e4f03b2d38c887164ca75f8b726ad5ef17117b73bbd7c24f985b707cc0ff
Sep 09 22:03:35.361739 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:35.361705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" event={"ID":"c16a3230-f960-49b8-b922-20e60bd0cdff","Type":"ContainerStarted","Data":"86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67"}
Sep 09 22:03:35.362206 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:35.361760    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" event={"ID":"c16a3230-f960-49b8-b922-20e60bd0cdff","Type":"ContainerStarted","Data":"44b7e4f03b2d38c887164ca75f8b726ad5ef17117b73bbd7c24f985b707cc0ff"}
Sep 09 22:03:37.372893 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:37.372860    2583 generic.go:356] "Generic (PLEG): container finished" podID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerID="86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67" exitCode=0
Sep 09 22:03:37.373493 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:37.372913    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" event={"ID":"c16a3230-f960-49b8-b922-20e60bd0cdff","Type":"ContainerDied","Data":"86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67"}
Sep 09 22:03:38.380371 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:38.380315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" event={"ID":"c16a3230-f960-49b8-b922-20e60bd0cdff","Type":"ContainerStarted","Data":"f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c"}
Sep 09 22:03:38.407927 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:38.407849    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" podStartSLOduration=2.264673437 podStartE2EDuration="4.407827899s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.143154462s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:03:38.406049714 +0000 UTC m=+4257.030523655" watchObservedRunningTime="2026-09-09 22:03:38.407827899 +0000 UTC m=+4257.032301838"
Sep 09 22:03:38.408997 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:38.408971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:44.409969 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:44.409935    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:03:44.411704 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:44.411668    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 09 22:03:54.409759 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:03:54.409703    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 09 22:04:04.410341 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:04:04.410293    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 09 22:04:14.409761 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:04:14.409716    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 09 22:04:22.003802 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:04:22.003765    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"]
Sep 09 22:04:37.029578 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:04:37.029498    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="main" containerID="cri-o://f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c" gracePeriod=45
Sep 09 22:05:22.680751 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.680727    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7f95b98795-vf8x6_c16a3230-f960-49b8-b922-20e60bd0cdff/main/0.log"
Sep 09 22:05:22.683288 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.683270    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:05:22.764281 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764236    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c16a3230-f960-49b8-b922-20e60bd0cdff-cabundle-cert\") pod \"c16a3230-f960-49b8-b922-20e60bd0cdff\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") "
Sep 09 22:05:22.764476 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764355    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c16a3230-f960-49b8-b922-20e60bd0cdff-tls-certs\") pod \"c16a3230-f960-49b8-b922-20e60bd0cdff\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") "
Sep 09 22:05:22.764476 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764403    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-dshm\") pod \"c16a3230-f960-49b8-b922-20e60bd0cdff\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") "
Sep 09 22:05:22.764476 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764440    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-home\") pod \"c16a3230-f960-49b8-b922-20e60bd0cdff\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") "
Sep 09 22:05:22.764476 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764462    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmrth\" (UniqueName: \"kubernetes.io/projected/c16a3230-f960-49b8-b922-20e60bd0cdff-kube-api-access-cmrth\") pod \"c16a3230-f960-49b8-b922-20e60bd0cdff\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") "
Sep 09 22:05:22.764726 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764505    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-kserve-provision-location\") pod \"c16a3230-f960-49b8-b922-20e60bd0cdff\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") "
Sep 09 22:05:22.764726 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764523    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-tmp-dir\") pod \"c16a3230-f960-49b8-b922-20e60bd0cdff\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") "
Sep 09 22:05:22.764726 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-model-cache\") pod \"c16a3230-f960-49b8-b922-20e60bd0cdff\" (UID: \"c16a3230-f960-49b8-b922-20e60bd0cdff\") "
Sep 09 22:05:22.764885 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764748    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16a3230-f960-49b8-b922-20e60bd0cdff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c16a3230-f960-49b8-b922-20e60bd0cdff" (UID: "c16a3230-f960-49b8-b922-20e60bd0cdff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:05:22.764885 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764828    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-home" (OuterVolumeSpecName: "home") pod "c16a3230-f960-49b8-b922-20e60bd0cdff" (UID: "c16a3230-f960-49b8-b922-20e60bd0cdff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:22.764995 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764927    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c16a3230-f960-49b8-b922-20e60bd0cdff-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.765034 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.764965    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-model-cache" (OuterVolumeSpecName: "model-cache") pod "c16a3230-f960-49b8-b922-20e60bd0cdff" (UID: "c16a3230-f960-49b8-b922-20e60bd0cdff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:22.765286 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.765260    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c16a3230-f960-49b8-b922-20e60bd0cdff" (UID: "c16a3230-f960-49b8-b922-20e60bd0cdff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:22.766962 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.766935    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-dshm" (OuterVolumeSpecName: "dshm") pod "c16a3230-f960-49b8-b922-20e60bd0cdff" (UID: "c16a3230-f960-49b8-b922-20e60bd0cdff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:22.767081 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.767002    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16a3230-f960-49b8-b922-20e60bd0cdff-kube-api-access-cmrth" (OuterVolumeSpecName: "kube-api-access-cmrth") pod "c16a3230-f960-49b8-b922-20e60bd0cdff" (UID: "c16a3230-f960-49b8-b922-20e60bd0cdff"). InnerVolumeSpecName "kube-api-access-cmrth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:05:22.767081 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.767049    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16a3230-f960-49b8-b922-20e60bd0cdff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c16a3230-f960-49b8-b922-20e60bd0cdff" (UID: "c16a3230-f960-49b8-b922-20e60bd0cdff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:05:22.788811 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.788775    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c16a3230-f960-49b8-b922-20e60bd0cdff" (UID: "c16a3230-f960-49b8-b922-20e60bd0cdff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:22.866070 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.865974    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-home\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.866070 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.866009    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmrth\" (UniqueName: \"kubernetes.io/projected/c16a3230-f960-49b8-b922-20e60bd0cdff-kube-api-access-cmrth\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.866070 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.866018    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.866070 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.866027    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-tmp-dir\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.866070 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.866035    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.866070 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.866054    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c16a3230-f960-49b8-b922-20e60bd0cdff-tls-certs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.866070 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.866061    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c16a3230-f960-49b8-b922-20e60bd0cdff-dshm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.902270 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.902242    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7f95b98795-vf8x6_c16a3230-f960-49b8-b922-20e60bd0cdff/main/0.log"
Sep 09 22:05:22.902647 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.902621    2583 generic.go:356] "Generic (PLEG): container finished" podID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerID="f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c" exitCode=137
Sep 09 22:05:22.902759 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.902664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" event={"ID":"c16a3230-f960-49b8-b922-20e60bd0cdff","Type":"ContainerDied","Data":"f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c"}
Sep 09 22:05:22.902759 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.902687    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"
Sep 09 22:05:22.902759 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.902712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6" event={"ID":"c16a3230-f960-49b8-b922-20e60bd0cdff","Type":"ContainerDied","Data":"44b7e4f03b2d38c887164ca75f8b726ad5ef17117b73bbd7c24f985b707cc0ff"}
Sep 09 22:05:22.902759 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.902738    2583 scope.go:117] "RemoveContainer" containerID="f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c"
Sep 09 22:05:22.912159 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.912135    2583 scope.go:117] "RemoveContainer" containerID="86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67"
Sep 09 22:05:22.920764 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.920742    2583 scope.go:117] "RemoveContainer" containerID="f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c"
Sep 09 22:05:22.921063 ip-10-0-140-132 kubenswrapper[2583]: E0909 22:05:22.921043    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c\": container with ID starting with f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c not found: ID does not exist" containerID="f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c"
Sep 09 22:05:22.921157 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.921077    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c"} err="failed to get container status \"f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c\": rpc error: code = NotFound desc = could not find container \"f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c\": container with ID starting with f008cb132214da5957c3ec4b81a59ff5805f34dd9da0368af9cd746209dfd26c not found: ID does not exist"
Sep 09 22:05:22.921157 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.921131    2583 scope.go:117] "RemoveContainer" containerID="86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67"
Sep 09 22:05:22.921417 ip-10-0-140-132 kubenswrapper[2583]: E0909 22:05:22.921397    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67\": container with ID starting with 86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67 not found: ID does not exist" containerID="86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67"
Sep 09 22:05:22.921458 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.921422    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67"} err="failed to get container status \"86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67\": rpc error: code = NotFound desc = could not find container \"86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67\": container with ID starting with 86c824f25443da74e1d5844e6b93bc58ca0b52d09e50d3f286c260a746e0ae67 not found: ID does not exist"
Sep 09 22:05:22.925937 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.925914    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"]
Sep 09 22:05:22.929575 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:22.929548    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-vf8x6"]
Sep 09 22:05:23.865442 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:05:23.865409    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" path="/var/lib/kubelet/pods/c16a3230-f960-49b8-b922-20e60bd0cdff/volumes"
Sep 09 22:06:20.273718 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:20.273679    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/0.log"
Sep 09 22:06:20.288244 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:20.288205    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/1.log"
Sep 09 22:06:20.301201 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:20.301172    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7579dfff88-lj6xx_20940e25-e1d8-41da-b973-55f00196c089/router/0.log"
Sep 09 22:06:21.080017 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:21.079982    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/0.log"
Sep 09 22:06:21.093707 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:21.093681    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/1.log"
Sep 09 22:06:21.109986 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:21.109952    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7579dfff88-lj6xx_20940e25-e1d8-41da-b973-55f00196c089/router/0.log"
Sep 09 22:06:21.891665 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:21.891624    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/0.log"
Sep 09 22:06:21.905708 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:21.905673    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/1.log"
Sep 09 22:06:21.919540 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:21.919517    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7579dfff88-lj6xx_20940e25-e1d8-41da-b973-55f00196c089/router/0.log"
Sep 09 22:06:22.829614 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:22.829544    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/0.log"
Sep 09 22:06:22.843451 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:22.843411    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/1.log"
Sep 09 22:06:22.858566 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:22.858535    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7579dfff88-lj6xx_20940e25-e1d8-41da-b973-55f00196c089/router/0.log"
Sep 09 22:06:24.397207 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:24.397171    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:24.407868 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:24.407846    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:26.014239 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:26.014195    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:26.032783 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:26.032755    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:27.613083 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:27.613050    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:27.629741 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:27.629716    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:29.182757 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:29.182721    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:29.194794 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:29.194768    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:30.754153 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:30.754121    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:30.765688 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:30.765665    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:32.319014 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:32.318984    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:32.330991 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:32.330965    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:33.890510 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:33.890472    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:33.902932 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:33.902903    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:35.452564 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:35.452521    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:35.465686 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:35.465658    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:37.062607 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:37.062554    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:37.074322 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:37.074295    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:38.636250 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:38.636206    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:38.647957 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:38.647928    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:40.189481 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:40.189446    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:40.201672 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:40.201639    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:41.735355 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:41.735320    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:41.747815 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:41.747787    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:43.297419 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:43.297387    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:43.310449 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:43.310420    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:44.857524 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:44.857485    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:44.869816 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:44.869791    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:46.399408 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:46.399362    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:46.412293 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:46.412255    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:47.975266 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:47.975233    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:47.992344 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:47.992317    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:49.541275 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:49.541229    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:49.553594 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:49.553561    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:51.074228 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:51.074193    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:51.085995 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:51.085975    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:52.638160 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:52.638099    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:52.649246 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:52.649216    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:54.178481 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:54.178442    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:54.190025 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:54.189993    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:55.711524 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:55.711490    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:55.741613 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:55.741581    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:57.316318 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:57.316265    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:57.331598 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:57.331568    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:06:58.887037 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:58.886998    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:06:58.899793 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:06:58.899767    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:07:00.403419 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:00.403376    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:07:00.416893 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:00.416862    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:07:01.978162 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:01.978117    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:07:01.990851 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:01.990821    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:07:08.520056 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:08.520025    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nrhbs_2685e808-1a0b-4fb0-b19a-876ead4ad42e/global-pull-secret-syncer/0.log"
Sep 09 22:07:08.656473 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:08.656442    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k6zp7_62c0bc3b-be18-47e4-bd35-d90287e22c4b/konnectivity-agent/0.log"
Sep 09 22:07:08.755541 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:08.755505    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-132.ec2.internal_9fdee4d29f0b1359d09bd37b4c77420b/haproxy/0.log"
Sep 09 22:07:14.201995 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:14.201951    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dzn25_ff3d4ace-6b31-4357-98ac-255b15446a5e/manager/0.log"
Sep 09 22:07:14.237854 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:14.237829    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw8rn_310ead78-ec2a-4d3d-a74d-12ebab9e711b/limitador/0.log"
Sep 09 22:07:15.662293 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:15.662264    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-hkqwt_ebea8244-c7f2-483d-83ab-3db65b3b770a/kube-state-metrics/0.log"
Sep 09 22:07:15.693792 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:15.693760    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-hkqwt_ebea8244-c7f2-483d-83ab-3db65b3b770a/kube-rbac-proxy-main/0.log"
Sep 09 22:07:15.719889 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:15.719860    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-hkqwt_ebea8244-c7f2-483d-83ab-3db65b3b770a/kube-rbac-proxy-self/0.log"
Sep 09 22:07:15.762525 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:15.762499    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6cc84b76b8-2nclq_de8df011-c523-45a1-8266-dc8cfe411749/metrics-server/0.log"
Sep 09 22:07:15.825257 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:15.825233    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-kj57r_d261ebf2-f8d5-4f8f-998f-dafb6640f066/monitoring-plugin/0.log"
Sep 09 22:07:15.961805 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:15.961776    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9c6k_025696b8-f9ab-44ea-a08a-060bc44b12ec/node-exporter/0.log"
Sep 09 22:07:15.984364 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:15.984334    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9c6k_025696b8-f9ab-44ea-a08a-060bc44b12ec/kube-rbac-proxy/0.log"
Sep 09 22:07:16.008459 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.008434    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9c6k_025696b8-f9ab-44ea-a08a-060bc44b12ec/init-textfile/0.log"
Sep 09 22:07:16.120910 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.120801    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-h7zvw_f8dfea5c-68bf-4e9f-b320-269f8cf7775b/kube-rbac-proxy-main/0.log"
Sep 09 22:07:16.142829 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.142799    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-h7zvw_f8dfea5c-68bf-4e9f-b320-269f8cf7775b/kube-rbac-proxy-self/0.log"
Sep 09 22:07:16.162857 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.162819    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-h7zvw_f8dfea5c-68bf-4e9f-b320-269f8cf7775b/openshift-state-metrics/0.log"
Sep 09 22:07:16.199527 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.199486    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93d5670-d259-4d7d-80f5-0c96dced55c3/prometheus/0.log"
Sep 09 22:07:16.216787 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.216768    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93d5670-d259-4d7d-80f5-0c96dced55c3/config-reloader/0.log"
Sep 09 22:07:16.239167 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.239078    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93d5670-d259-4d7d-80f5-0c96dced55c3/thanos-sidecar/0.log"
Sep 09 22:07:16.262480 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.262441    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93d5670-d259-4d7d-80f5-0c96dced55c3/kube-rbac-proxy-web/0.log"
Sep 09 22:07:16.283633 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.283575    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93d5670-d259-4d7d-80f5-0c96dced55c3/kube-rbac-proxy/0.log"
Sep 09 22:07:16.305077 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.305043    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93d5670-d259-4d7d-80f5-0c96dced55c3/kube-rbac-proxy-thanos/0.log"
Sep 09 22:07:16.329038 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.329013    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93d5670-d259-4d7d-80f5-0c96dced55c3/init-config-reloader/0.log"
Sep 09 22:07:16.365360 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.365326    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wlbb2_8b35118e-dda0-43f2-bf20-312ebbd27efa/prometheus-operator/0.log"
Sep 09 22:07:16.383268 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.383195    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wlbb2_8b35118e-dda0-43f2-bf20-312ebbd27efa/kube-rbac-proxy/0.log"
Sep 09 22:07:16.527384 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.527354    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694bd65f4d-gbcwm_6508432d-5e5c-4400-83a0-dad1cfe1c39f/thanos-query/0.log"
Sep 09 22:07:16.550680 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.550659    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694bd65f4d-gbcwm_6508432d-5e5c-4400-83a0-dad1cfe1c39f/kube-rbac-proxy-web/0.log"
Sep 09 22:07:16.574582 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.574554    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694bd65f4d-gbcwm_6508432d-5e5c-4400-83a0-dad1cfe1c39f/kube-rbac-proxy/0.log"
Sep 09 22:07:16.601414 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.601390    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694bd65f4d-gbcwm_6508432d-5e5c-4400-83a0-dad1cfe1c39f/prom-label-proxy/0.log"
Sep 09 22:07:16.624470 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.624443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694bd65f4d-gbcwm_6508432d-5e5c-4400-83a0-dad1cfe1c39f/kube-rbac-proxy-rules/0.log"
Sep 09 22:07:16.652162 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:16.652064    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694bd65f4d-gbcwm_6508432d-5e5c-4400-83a0-dad1cfe1c39f/kube-rbac-proxy-metrics/0.log"
Sep 09 22:07:17.835378 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:17.835345    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-pkjg7_eb34e84e-d11f-4ea2-97f2-e79ada097ce7/networking-console-plugin/0.log"
Sep 09 22:07:19.152665 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.152630    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"]
Sep 09 22:07:19.153100 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.152995    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="storage-initializer"
Sep 09 22:07:19.153100 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.153014    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="storage-initializer"
Sep 09 22:07:19.153100 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.153041    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="main"
Sep 09 22:07:19.153100 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.153049    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="main"
Sep 09 22:07:19.153349 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.153173    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c16a3230-f960-49b8-b922-20e60bd0cdff" containerName="main"
Sep 09 22:07:19.156486 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.156469    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.159077 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.159043    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tw9dj\"/\"default-dockercfg-st6zb\""
Sep 09 22:07:19.159077 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.159045    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tw9dj\"/\"openshift-service-ca.crt\""
Sep 09 22:07:19.160616 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.160595    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tw9dj\"/\"kube-root-ca.crt\""
Sep 09 22:07:19.165618 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.165596    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"]
Sep 09 22:07:19.204491 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.204455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-sys\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.204694 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.204500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-lib-modules\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.204694 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.204522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-podres\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.204694 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.204542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2756\" (UniqueName: \"kubernetes.io/projected/423837df-b2be-4267-928d-5e72cfd6ae8c-kube-api-access-q2756\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.204694 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.204591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-proc\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.305718 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.305681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-podres\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.305718 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.305719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2756\" (UniqueName: \"kubernetes.io/projected/423837df-b2be-4267-928d-5e72cfd6ae8c-kube-api-access-q2756\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.305969 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.305754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-proc\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.305969 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.305852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-sys\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.305969 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.305861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-podres\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.305969 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.305905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-lib-modules\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.305969 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.305958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-sys\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.306198 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.305960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-proc\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.306198 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.306026    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/423837df-b2be-4267-928d-5e72cfd6ae8c-lib-modules\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.314976 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.314946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2756\" (UniqueName: \"kubernetes.io/projected/423837df-b2be-4267-928d-5e72cfd6ae8c-kube-api-access-q2756\") pod \"perf-node-gather-daemonset-hvvsc\" (UID: \"423837df-b2be-4267-928d-5e72cfd6ae8c\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.467853 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.467746    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:19.739859 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:19.739807    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"]
Sep 09 22:07:19.742215 ip-10-0-140-132 kubenswrapper[2583]: W0909 22:07:19.742185    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod423837df_b2be_4267_928d_5e72cfd6ae8c.slice/crio-33e31c5923ca6ff17ee8fc037fea30f0edffe58c24c14274d92042fc900115cc WatchSource:0}: Error finding container 33e31c5923ca6ff17ee8fc037fea30f0edffe58c24c14274d92042fc900115cc: Status 404 returned error can't find the container with id 33e31c5923ca6ff17ee8fc037fea30f0edffe58c24c14274d92042fc900115cc
Sep 09 22:07:20.466011 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:20.465967    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc" event={"ID":"423837df-b2be-4267-928d-5e72cfd6ae8c","Type":"ContainerStarted","Data":"2d43f4f464024e7b941e10b151b3fa037fcae8e5b3e56d7fd1f6890644bbd547"}
Sep 09 22:07:20.466471 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:20.466025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc" event={"ID":"423837df-b2be-4267-928d-5e72cfd6ae8c","Type":"ContainerStarted","Data":"33e31c5923ca6ff17ee8fc037fea30f0edffe58c24c14274d92042fc900115cc"}
Sep 09 22:07:20.466471 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:20.466127    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:20.484460 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:20.484397    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc" podStartSLOduration=1.484378605 podStartE2EDuration="1.484378605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:20.483803096 +0000 UTC m=+4479.108277037" watchObservedRunningTime="2026-09-09 22:07:20.484378605 +0000 UTC m=+4479.108852552"
Sep 09 22:07:20.594608 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:20.594578    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rjl4w_17fc2904-8ec6-4d26-8f39-14abdbe1a1cc/dns/0.log"
Sep 09 22:07:20.615351 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:20.615321    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rjl4w_17fc2904-8ec6-4d26-8f39-14abdbe1a1cc/kube-rbac-proxy/0.log"
Sep 09 22:07:20.713932 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:20.713896    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mmcfd_b5c2c926-9449-4539-b439-06fb5d910856/dns-node-resolver/0.log"
Sep 09 22:07:21.353735 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:21.353704    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mxtw2_c468bb92-a81e-4d78-88c0-ae85f7c69561/node-ca/0.log"
Sep 09 22:07:22.414621 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:22.414590    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/0.log"
Sep 09 22:07:22.429041 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:22.429003    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/1.log"
Sep 09 22:07:22.455017 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:22.454992    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7579dfff88-lj6xx_20940e25-e1d8-41da-b973-55f00196c089/router/0.log"
Sep 09 22:07:23.018352 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:23.018321    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7plsw_393acb6a-39f1-46c0-9c3a-29f50af68eff/serve-healthcheck-canary/0.log"
Sep 09 22:07:23.642291 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:23.642264    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7hwww_394b88bf-e239-4ffd-8c70-4c633a0292db/kube-rbac-proxy/0.log"
Sep 09 22:07:23.665305 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:23.665277    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7hwww_394b88bf-e239-4ffd-8c70-4c633a0292db/exporter/0.log"
Sep 09 22:07:23.689911 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:23.689879    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7hwww_394b88bf-e239-4ffd-8c70-4c633a0292db/extractor/0.log"
Sep 09 22:07:26.481565 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:26.481538    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-hvvsc"
Sep 09 22:07:26.878816 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:26.878778    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85b69767b4-7dc9x_59bb1174-2459-4493-bfdc-44f6232ce83d/manager/0.log"
Sep 09 22:07:26.935071 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:26.935039    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9bdfv_e340f7b6-75b1-4796-80f8-699db99dc8f8/openshift-lws-operator/0.log"
Sep 09 22:07:28.656449 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:28.656419    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/0.log"
Sep 09 22:07:28.669094 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:28.669070    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/1.log"
Sep 09 22:07:28.687071 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:28.687049    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7579dfff88-lj6xx_20940e25-e1d8-41da-b973-55f00196c089/router/0.log"
Sep 09 22:07:29.970994 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:29.970897    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/0.log"
Sep 09 22:07:29.984798 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:29.984769    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-dnv9k_d138945c-aa5e-4fb7-86c5-48e959a29350/istio-proxy/1.log"
Sep 09 22:07:29.999810 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:29.999783    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7579dfff88-lj6xx_20940e25-e1d8-41da-b973-55f00196c089/router/0.log"
Sep 09 22:07:35.865915 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:35.865884    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p7m7l_c58b4f94-071e-400c-bc9b-e72bb86a324f/kube-multus-additional-cni-plugins/0.log"
Sep 09 22:07:35.889872 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:35.889843    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p7m7l_c58b4f94-071e-400c-bc9b-e72bb86a324f/egress-router-binary-copy/0.log"
Sep 09 22:07:35.910397 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:35.910375    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p7m7l_c58b4f94-071e-400c-bc9b-e72bb86a324f/cni-plugins/0.log"
Sep 09 22:07:35.930720 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:35.930686    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p7m7l_c58b4f94-071e-400c-bc9b-e72bb86a324f/bond-cni-plugin/0.log"
Sep 09 22:07:35.951335 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:35.951302    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p7m7l_c58b4f94-071e-400c-bc9b-e72bb86a324f/routeoverride-cni/0.log"
Sep 09 22:07:35.974165 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:35.974141    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p7m7l_c58b4f94-071e-400c-bc9b-e72bb86a324f/whereabouts-cni-bincopy/0.log"
Sep 09 22:07:35.996686 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:35.996660    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p7m7l_c58b4f94-071e-400c-bc9b-e72bb86a324f/whereabouts-cni/0.log"
Sep 09 22:07:36.036324 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:36.036300    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m92df_02275490-d835-48ad-836f-4eb605bd4d0f/kube-multus/0.log"
Sep 09 22:07:36.062469 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:36.062449    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-29hf4_d43fba51-6814-4c9e-a5c8-a1dd796b7386/network-metrics-daemon/0.log"
Sep 09 22:07:36.081942 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:36.081913    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-29hf4_d43fba51-6814-4c9e-a5c8-a1dd796b7386/kube-rbac-proxy/0.log"
Sep 09 22:07:37.212486 ip-10-0-140-132 kubenswrapper[2583]: I0909 22:07:37.212458    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8nfkh_0f8b5d92-d112-4bbc-a824-b4a96710d5db/ovn-controller/0.log"